Starlake

Installation Methods

🖥️ Linux/MacOS

Prerequisites

• Java 17+ installed

  # Verify Java version
  java -version

• If needed, install Java from Oracle JDK or OpenJDK

Installation

One-line installation script:

bash <(curl https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.sh)

The script will create the following directory structure:

starlake/
├── bin/
│   ├── deps/      # Extra dependencies (e.g., Oracle client)
│   ├── sl         # Starlake CLI
│   └── spark      # Embedded Spark runtime

🪟 Windows

Prerequisites

• Java 17+ installed

  # Verify Java version
  java -version

• PowerShell 5.0 or later
• Administrator privileges

Installation

Run in PowerShell as administrator:

Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.ps1").Content

🐳 Docker

Prerequisites

• Docker installed and running
• No other dependencies required - everything is included in the container

Installation

The fastest way to get started with Starlake:

# Pull the latest stable image
docker pull starlakeai/starlake:latest

# Verify installation
docker run -it starlakeai/starlake:latest help

Custom Build Instructions

Build a specific version or branch:

# Clone repository
git clone git@github.com:starlake-ai/starlake.git
cd starlake

# Build with specific version
docker build -t starlakeai/starlake:VERSION \
    --build-arg SL_VERSION=1.2.0 .

Running Starlake with Docker

Basic Usage

# Mount current directory and run Starlake
docker run -it \
    -e SL_ROOT=/starlake \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest <command>

# Example: Run help command
docker run -it \
    -e SL_ROOT=/starlake \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest help

AWS Integration

# Run with AWS credentials
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
    -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
    -e AWS_REGION=$AWS_REGION \
    starlakeai/starlake:latest <command>

GCP Integration

# Run with GCP credentials
docker run -it \
    -v $(pwd):/starlake \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    -e SL_ROOT=/starlake \
    -e GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json \
    starlakeai/starlake:latest <command>

Azure Integration

# Run with Azure credentials
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AZURE_TENANT_ID=$AZURE_TENANT_ID \
    -e AZURE_CLIENT_ID=$AZURE_CLIENT_ID \
    -e AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET \
    starlakeai/starlake:latest <command>

Advanced Options

# Run with custom configuration
docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e SL_ENV=production \
    -e SL_METRICS_ACTIVE=true \
    -e JAVA_OPTS="-Xmx4g" \
    -e SPARK_DRIVER_MEMORY=4g \
    starlakeai/starlake:latest <command>

Common environment variables:

• SL_ROOT: Project root directory inside container
• SL_ENV: Environment (dev, test, prod)
• SL_METRICS_ACTIVE: Enable/disable metrics collection
• JAVA_OPTS: JVM options
• SPARK_DRIVER_MEMORY: Spark driver memory allocation

Mount Points

Always mount your project directory to /starlake in the container to ensure proper file access.

Cloud Credentials

For production environments, consider using mounted credential files or secret management services instead of environment variables.

Development Tools

VS Code Extension

Prerequisites

• Visual Studio Code installed

Installation

Enhance your development experience with our official VS Code extension:

1. Open VS Code
2. Go to Extensions (Ctrl+Shift+X)
3. Search for "Starlake"
4. Install "Starlake Data Pipeline Tools"

Features:

• Syntax highlighting for Starlake configurations
• Schema validation
• SQL transformation snippets
• Pipeline visualization

Verify Installation

Test your installation:

starlake --version

You should see output like:

Starlake Version 1.3.4

Next Steps

You may start the GUI with:

starlake serve

and point your browser to http://localhost:9000.

To start on a different port set the SL_API_HTTP_PORT environment variable:

SL_API_HTTP_PORT=11000 starlake serve

To access the GUI from a remote machine, set the SL_API_DOMAIN environment variable to your server's domain or IP address:

SL_API_DOMAIN=your.domain.com starlake serve

The exhaustive list of configuration options can be found in the api-env-vars.md file. or continue with the CLI by following the tutorials below:

• Create your first project
• Load your first dataset
• Transform your data

Troubleshooting

Common installation issues and solutions:

1. Java Version Mismatch

   # Check Java version
   java -version
   
   # Install correct version if needed

2. Permission Issues

   # Fix permissions on Linux/MacOS
   chmod +x bin/starlake

3. Docker Issues

   # Verify Docker installation
   docker --version
   
   # Test Docker permissions
   docker run hello-world

Extra Dependencies

Any additional libraries (e.g., Oracle client) should be copied to the bin/deps folder.