Pathway

What is Pathway?

Pathway is a software framework developed to address the industry challenge of lacking software infrastructure capable of automated reasoning on data streams in real-time. It enables the creation of real-time data products by allowing software engineers to focus on code logic without worrying about changing data inputs. Pathway's reactive design ensures that data updates are managed seamlessly, resulting in data products that are easy to develop, maintain, and run efficiently on top of its engine. It streamlines the ML/AI project lifecycle from prototype to production, supporting various deployment options and providing high-speed updates from live data sources like cloud folders, databases, APIs, and more. Pathway's features include streamlined cloud-native development, fast and synchronized data processing, deployment flexibility, customizable code templates, and interaction with external APIs and models like LLMs.

Who created Pathway?

Pathway was created by a team of individuals with backgrounds from the best AI labs, including ex-Googlers, top competitive programmers, and experts in Distributed Systems, Machine Learning, and Complexity Science. The founder, Zuzanna, completed her PhD in forecasting maritime trade and identified a need for applying Machine Learning to data streams in logistics. She was joined by Jan, who had experience at Google Brain, along with Adrian and Claire. Together, they developed Pathway to address the industry's lack of software infrastructure for real-time data processing.

Who is Pathway for?

• Software engineers
• Python Developers

How to use Pathway?

To use Pathway effectively, follow these steps:

1. Setting Up Data Ingestion: Easily connect and sync data from over 300 sources for vector search, real-time features, and anomaly alerts. Pathway automatically synchronizes data for accurate AI insights from connected documents and tables.

2. App Templates Setup Guide: Pathway offers app templates and a setup guide to create real-time data fetching applications without the need for complex databases or stacks.

3. Streamlined Development: Built with Rust for Python developers, Pathway streamlines the ML/AI project lifecycle from prototype to production. It supports deployments across different environments like local, notebook, and scaled containers.

4. Connect Live Data Sources: Access live data from sources like SharePoint, Google Drive, S3, Delta Tables, Kafka, databases, and over 300 APIs such as Salesforce and Hubspot. Pathway's high-speed engine provides real-time updates, scalability, and cost-efficient computing.

5. Deployment Flexibility: Own your deployment using Docker or Kubernetes, on-premises or cloud, avoiding multiple databases and compute engines. Simplify your setup and gain full control over your data pathways.

6. Customization and Integration: Choose from ready-to-use Pathway templates tailored for different industries and data types. Customize using Pathway's library, connectors, and integrations for safe interaction with LLMs and external APIs.

7. Reactive Design: Pathway allows Python developers to easily build real-time data products at scale, focusing on code logic without worrying about data updates. The framework manages data processing seamlessly for up-to-date BI dashboards.

By following these steps, users can leverage Pathway's capabilities efficiently in developing real-time data applications for a variety of use cases.