go-oversync

Getting Started

This guide walks you through creating your own simple HTTP server with go-oversync integration. You’ll build a complete sync-enabled server with users and posts tables from scratch.

Prerequisites

Core Concepts

Before building your sync server, it’s important to understand the key components that make go-oversync work:

User ID

Device ID (aka Source ID)

Why Device IDs are Essential:

  1. Idempotency Protection
    • Prevents duplicate processing when network requests are retried
    • Server uses (User ID + Device ID + Source Change ID) as unique key
    • Same change uploaded multiple times is safely ignored
  2. Echo Suppression
    • Devices don’t download their own changes back
    • Prevents infinite sync loops where Device A uploads → Device A downloads same change
    • Server excludes changes from requesting Device ID during downloads
  3. Conflict Resolution
    • Helps identify which device made which changes
    • Enables “last writer wins” or custom conflict resolution strategies
    • Audit trail for debugging sync issues
  4. Hydration Recovery
    • When app is reinstalled, new Device ID triggers full hydration
    • Server sends all user’s data (include_self=true) for complete recovery
    • Prevents partial sync state after fresh installation

Server Version

Server ID (Sequence)

Source Change ID

Sync Tables

go-oversync creates these tables automatically to manage sync metadata:

Your business tables (like users and posts) remain completely unchanged. Well, almost unchanged, we migrate foreign keys to deferred to support deferred batch ordering.

How Sync Works

Here’s how these components work together in a typical sync flow:

Upload Flow:

  1. Client makes changes to local SQLite database
  2. Client sends changes with: User ID, Device ID, Source Change ID, Server Version
  3. Server checks: “Is this User ID + Device ID + Source Change ID already processed?” (idempotency)
  4. Server checks: “Does the Server Version match current version?” (conflict detection)
  5. If valid: Server applies change, increments Server Version, assigns Server ID
  6. Server responds with new Server Version or conflict details

Download Flow:

  1. Client requests: “Give me changes after Server ID X for my User ID”
  2. Server returns: Ordered list of changes (excluding client’s own Device ID)
  3. Client applies changes locally and updates its “last seen Server ID”
  4. Process repeats to stay in sync

Multi-Device Example:

Step 1: Set Up PostgreSQL

Start PostgreSQL and create a database:

docker run --name oversync-pg -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:16
createdb my_sync_app  # or use psql to create the DB

Step 2: Create Your Project

Initialize a new Go project:

mkdir my-sync-server
cd my-sync-server
go mod init my-sync-server
go get github.com/mobiletoly/go-oversync
go get github.com/jackc/pgx/v5/pgxpool

Step 3: Create Your Business Tables

Create a SQL file schema.sql to define your business tables:

-- Create business schema
CREATE SCHEMA IF NOT EXISTS business;

-- Users table
CREATE TABLE IF NOT EXISTS business.users (
    id UUID PRIMARY KEY,
    name TEXT NOT NULL,
    email TEXT UNIQUE NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Posts table with foreign key to users
CREATE TABLE IF NOT EXISTS business.posts (
    id UUID PRIMARY KEY,
    author_id UUID NOT NULL REFERENCES business.users(id) ON DELETE CASCADE,
    title TEXT NOT NULL,
    content TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

Step 4: Create Your Server

Create main.go with your HTTP server:

package main

import (
    "context"
    "database/sql"
    "encoding/json"
    "log"
    "log/slog"
    "net/http"
    "os"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"
    "github.com/mobiletoly/go-oversync/oversync"
    _ "github.com/jackc/pgx/v5/stdlib"
)

func main() {
    ctx := context.Background()
    logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

    // Connect to PostgreSQL
    databaseURL := "postgres://postgres:postgres@localhost:5432/my_sync_app?sslmode=disable"

    pool, err := pgxpool.New(ctx, databaseURL)
    if err != nil {
        log.Fatal("Failed to connect to database:", err)
    }
    defer pool.Close()

    // Initialize business tables
    if err := initializeBusinessTables(ctx, pool); err != nil {
        log.Fatal("Failed to initialize business tables:", err)
    }

    // Configure go-oversync
    cfg := &oversync.ServiceConfig{
        MaxSupportedSchemaVersion: 1,  // Version for client compatibility checks
        AppName: "my-sync-app",
        RegisteredTables: []oversync.RegisteredTable{
            // Only register tables you want to sync
            {Schema: "business", Table: "users"},
            {Schema: "business", Table: "posts"},
        },
    }

    // Create sync service
    svc, err := oversync.NewSyncService(pool, cfg, logger)
    if err != nil {
        log.Fatal("Failed to create sync service:", err)
    }

    // Set up authentication (using JWT for this example)
    // JWT will contain 'sub' (User ID) and 'did' (Device ID) claims
    jwtSecret := "your-secret-key-change-in-production"
    jwtAuth := oversync.NewJWTAuth(jwtSecret)
	
    // Create sync handlers. Here we pass the JWT authenticator directly, but you can pass 
    // your own authenticator that implements the oversync.ClientAuthenticator interface
    h := oversync.NewSyncHandlers(svc, jwtAuth, logger)

    mux := http.NewServeMux()
    // Authentication endpoint to return JWT token to the client for sync
    mux.HandleFunc("POST /signin", func(w http.ResponseWriter, r *http.Request) {
        handleSignIn(w, r, jwtAuth)
    })
    // If needed, add other authentication endpoints, e.g. to refresh tokens or to sign out

    // Sync endpoints (protected by JWT authentication)
    mux.Handle("POST /sync/upload", jwtAuth.Middleware(http.HandlerFunc(h.HandleUpload)))
    mux.Handle("GET /sync/download", jwtAuth.Middleware(http.HandlerFunc(h.HandleDownload)))

    logger.Info("Server starting on :8080")
    if err := http.ListenAndServe(":8080", mux); err != nil {
        log.Fatal("Server failed:", err)
    }
}

func initializeBusinessTables(ctx context.Context, pool *pgxpool.Pool) error {
    // Read and execute schema.sql
    schema := `
        CREATE SCHEMA IF NOT EXISTS business;

        CREATE TABLE IF NOT EXISTS business.users (
            id UUID PRIMARY KEY,
            name TEXT NOT NULL,
            email TEXT UNIQUE NOT NULL,
            created_at TIMESTAMPTZ DEFAULT NOW()
        );

        CREATE TABLE IF NOT EXISTS business.posts (
            id UUID PRIMARY KEY,
            author_id UUID NOT NULL REFERENCES business.users(id) ON DELETE CASCADE,
            title TEXT NOT NULL,
            content TEXT,
            created_at TIMESTAMPTZ DEFAULT NOW()
        );
    `

    _, err := pool.Exec(ctx, schema)
    return err
}

type SignInRequest struct {
    Username string `json:"username"`
    Password string `json:"password"`
    DeviceID string `json:"device_id"`
}

type SignInResponse struct {
    Token    string `json:"token"`
    UserID   string `json:"user_id"`
    DeviceID string `json:"device_id"`
}

func handleSignIn(w http.ResponseWriter, r *http.Request, jwtAuth *oversync.JWTAuth) {
    var req SignInRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "Invalid JSON", http.StatusBadRequest)
        return
    }

    // Validate req.Username and req.Password, you can allow empty password for demo
    // ...

    // Generate User ID based on username (in production, this comes from your user database)
    // This becomes the JWT 'sub' claim and provides complete data isolation per user
    userID := "user-" + req.Username

    // Device ID comes from client (usually generated once per app installation)
    // This becomes the JWT 'did' claim and enables idempotency + echo suppression
    deviceID := req.DeviceID

    // Generate JWT token with User ID and Device ID (1 hour expiry)
    // go-oversync will extract these from the JWT for all sync operations
    token, err := jwtAuth.GenerateToken(userID, deviceID, 1*time.Hour)
    if err != nil {
        http.Error(w, "Failed to generate token", http.StatusInternalServerError)
        return
    }

    // Return the token and IDs for client reference
    resp := SignInResponse{
        Token:    token,
        UserID:   userID,
        DeviceID: deviceID,
    }
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(resp)
}

Note on authentication

For convenience, go-oversync provides a small helper package at github.com/mobiletoly/go-oversync/oversync/jwt with a ready-to-use JWT middleware and token utilities. It expects sub (user id) and did (device id) claims and works out of the box with the sync handlers.

You are not required to use this package: you can plug in your own authorization and identity extraction (sessions, custom JWTs, API keys, etc.). Just ensure your middleware sets user and device identity consistently so uploads/downloads are scoped correctly and idempotency works as designed.

Step 5: Run Your Server

Start your server:

go run main.go

Your server will:

Step 6: Test Your Sync Server

Create a Test Client

Create a new file client_test.go to test your sync server using the oversqlite client:

package main

import (
    "context"
    "database/sql"
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "os"
    "strings"

    "github.com/google/uuid"
    "github.com/mobiletoly/go-oversync/oversqlite"
    _ "github.com/mattn/go-sqlite3"
)

type SignInRequest struct {
    Username string `json:"username"`
    Password string `json:"password"`
    DeviceID string `json:"device_id"`
}

type SignInResponse struct {
    Token    string `json:"token"`
    UserID   string `json:"user_id"`
    DeviceID string `json:"device_id"`
}

func main() {
    ctx := context.Background()

    // Create SQLite database
    db, err := sql.Open("sqlite3", "test_client.db")
    if err != nil {
        log.Fatal("Failed to open SQLite database:", err)
    }
    defer db.Close()

    // Create business tables in SQLite
    if err := createClientTables(db); err != nil {
        log.Fatal("Failed to create client tables:", err)
    }

    // Generate or retrieve Device ID (in production, store this persistently)
    deviceID := getOrCreateDeviceID()

    // Sign in to get token (contains User ID and Device ID)
    userID, _, token, err := signIn("jane", "any-password", deviceID)
    if err != nil {
        log.Fatal("Failed to sign in:", err)
    }
    fmt.Printf("Signed in as %s with device %s\n", userID, deviceID)

    // Configure oversqlite client for the tables we want to sync
    cfg := oversqlite.DefaultConfig("business", []oversqlite.SyncTable{
        {TableName: "users"},
        {TableName: "posts"},
    })

    // Token provider function - oversqlite will call this to get auth tokens
    tokenProvider := func(ctx context.Context) (string, error) {
        return token, nil
    }

    // Create sync client with User ID and Device ID
    // These must match what's in the JWT token for authentication to work
    client, err := oversqlite.NewClient(
        db,                      // Local SQLite database
        "http://localhost:8080", // Your sync server URL
        userID,                  // User ID for data isolation
        deviceID,                // Device ID for idempotency and echo suppression
        tokenProvider,           // Function to get auth tokens
        cfg,                     // Table configuration
    )
    if err != nil {
        log.Fatal("Failed to create sync client:", err)
    }

    // Test the sync flow
    if err := testSyncFlow(ctx, db, client); err != nil {
        log.Fatal("Sync test failed:", err)
    }

    fmt.Println("✅ Sync test completed successfully!")
}

Now add the helper functions to complete the client:

func signIn(username, password, deviceID string) (string, string, string, error) {
    reqBody := SignInRequest{
        Username: username,
        Password: password,
        DeviceID: deviceID,
    }
    jsonData, err := json.Marshal(reqBody)
    if err != nil {
        return "", "", "", err
    }

    resp, err := http.Post("http://localhost:8080/signin", "application/json", strings.NewReader(string(jsonData)))
    if err != nil {
        return "", "", "", err
    }
    defer resp.Body.Close()

    var signInResp SignInResponse
    if err := json.NewDecoder(resp.Body).Decode(&signInResp); err != nil {
        return "", "", "", err
    }

    return signInResp.UserID, signInResp.DeviceID, signInResp.Token, nil
}

func createClientTables(db *sql.DB) error {
    schema := `
        CREATE TABLE IF NOT EXISTS users (
            id TEXT PRIMARY KEY,
            name TEXT NOT NULL,
            email TEXT UNIQUE NOT NULL,
            created_at DATETIME DEFAULT CURRENT_TIMESTAMP
        );

        CREATE TABLE IF NOT EXISTS posts (
            id TEXT PRIMARY KEY,
            author_id TEXT NOT NULL,
            title TEXT NOT NULL,
            content TEXT,
            created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
            FOREIGN KEY (author_id) REFERENCES users(id) ON DELETE CASCADE
        );
    `
    _, err := db.Exec(schema)
    return err
}

func testSyncFlow(ctx context.Context, db *sql.DB, client *oversqlite.Client) error {
    // Step 1: Insert a user locally (normal SQLite INSERT)
    // oversqlite automatically tracks this change via triggers
    userID := uuid.New().String()
    _, err := db.ExecContext(ctx, `
        INSERT INTO users (id, name, email)
        VALUES (?, ?, ?)
    `, userID, "Jane Doe", "jane@example.com")
    if err != nil {
        return fmt.Errorf("failed to insert user: %w", err)
    }
    fmt.Println("Created user locally (Server Version: 0, Source Change ID: auto-assigned)")

    // Step 2: Upload changes to server
    // Client sends: User ID, Device ID, Source Change ID, Server Version 0
    // Server responds with: new Server Version 1, Server ID assigned
    if err := client.UploadOnce(ctx); err != nil {
        return fmt.Errorf("failed to upload: %w", err)
    }
    fmt.Println("Uploaded user to server (Server assigned Server Version 1)")

    // Step 3: Insert a post locally with foreign key to user
    postID := uuid.New().String()
    _, err = db.ExecContext(ctx, `
        INSERT INTO posts (id, author_id, title, content)
        VALUES (?, ?, ?, ?)
    `, postID, userID, "My First Post", "Hello, sync world!")
    if err != nil {
        return fmt.Errorf("failed to insert post: %w", err)
    }
    fmt.Println("Created post locally (Server Version: 0, new Source Change ID)")

    // Step 4: Upload the post
    // Server validates foreign key exists, assigns new Server ID and Server Version
    if err := client.UploadOnce(ctx); err != nil {
        return fmt.Errorf("failed to upload post: %w", err)
    }
    fmt.Println("Uploaded post to server (Server assigned Server Version 1)")

    // Step 5: Download changes (simulating another device)
    // Requests: "Give me changes after my last seen Server ID"
    // Server returns: Changes from other devices (excludes our Device ID)
    downloaded, hasMore, err := client.DownloadOnce(ctx, 100)
    if err != nil {
        return fmt.Errorf("failed to download: %w", err)
    }
    fmt.Printf("Downloaded %d changes from other devices (has_more: %t)\n", downloaded, hasMore)

    // Step 6: Verify data exists locally
    var userCount, postCount int
    db.QueryRowContext(ctx, "SELECT COUNT(*) FROM users").Scan(&userCount)
    db.QueryRowContext(ctx, "SELECT COUNT(*) FROM posts").Scan(&postCount)

    fmt.Printf("Local database: %d users, %d posts\n", userCount, postCount)
    fmt.Println("All sync metadata is stored in separate _sync_* tables")
    return nil
}

// getOrCreateDeviceID demonstrates Device ID best practices
// In production, store this in secure, persistent storage
func getOrCreateDeviceID() string {
    // For this example, we'll use a simple file-based approach
    // In mobile apps, use Keychain (iOS) or EncryptedSharedPreferences (Android)
    const deviceIDFile = "device_id.txt"
    // Try to read existing Device ID
    if data, err := os.ReadFile(deviceIDFile); err == nil {
        deviceID := strings.TrimSpace(string(data))
        if deviceID != "" {
            fmt.Printf("Using existing Device ID: %s\n", deviceID)
            return deviceID
        }
    }

    // Generate new Device ID
    deviceID := "desktop-" + uuid.New().String()
    // Store it persistently
    if err := os.WriteFile(deviceIDFile, []byte(deviceID), 0600); err != nil {
        log.Printf("Warning: Could not save Device ID: %v", err)
    }

    fmt.Printf("Generated new Device ID: %s\n", deviceID)
    return deviceID
}

Run the Test Client

Make sure your server is running:

go run main.go

This demonstrates:

  1. Authentication — Getting a JWT token from your /signin endpoint
  2. Local data creation — Inserting records into SQLite
  3. Upload sync — Sending changes to PostgreSQL server
  4. Download sync — Receiving changes from other devices
  5. Automatic change tracking — oversqlite handles all sync metadata

SQLite Primary Key Requirements

For go-oversync to work with your SQLite database, all primary key columns must be TEXT (or BLOB) type containing UUID values. This is a fundamental requirement for the sync system to properly track and identify records across devices.

Supported Primary Key Formats

Option 1: TEXT with UUID strings

CREATE TABLE users (
    id TEXT PRIMARY KEY,           -- UUID as string: "550e8400-e29b-41d4-a716-446655440000"
    name TEXT NOT NULL,
    email TEXT UNIQUE NOT NULL
);

CREATE TABLE posts (
    id TEXT PRIMARY KEY,           -- UUID as string
    author_id TEXT NOT NULL,       -- Foreign key also as UUID string
    title TEXT NOT NULL,
    content TEXT,
    FOREIGN KEY (author_id) REFERENCES users(id)
);

Option 2: BLOB with random bytes

CREATE TABLE users (
    id BLOB PRIMARY KEY,           -- 16-byte random BLOB
    name TEXT NOT NULL,
    email TEXT UNIQUE NOT NULL
) WITHOUT ROWID;

CREATE TABLE posts (
    id BLOB PRIMARY KEY,           -- 16-byte random BLOB
    author_id BLOB NOT NULL,       -- Foreign key as BLOB
    title TEXT NOT NULL,
    content TEXT,
    FOREIGN KEY (author_id) REFERENCES users(id)
) WITHOUT ROWID;

-- Insert with SQLite's randomblob() function
INSERT INTO users (id, name, email) VALUES (randomblob(16), 'Alice', 'alice@example.com');
INSERT INTO users (id, name, email) VALUES (randomblob(16), 'Bob', 'bob@example.com');

Option 2b: BLOB with randomblob() as default value

CREATE TABLE users (
    id BLOB PRIMARY KEY DEFAULT (randomblob(16)),  -- Auto-generate random 16-byte BLOB
    name TEXT NOT NULL,
    email TEXT UNIQUE NOT NULL
) WITHOUT ROWID;

CREATE TABLE posts (
    id BLOB PRIMARY KEY DEFAULT (randomblob(16)),  -- Auto-generate random 16-byte BLOB
    author_id BLOB NOT NULL,                       -- Foreign key as BLOB
    title TEXT NOT NULL,
    content TEXT,
    FOREIGN KEY (author_id) REFERENCES users(id)
) WITHOUT ROWID;

-- Insert without specifying ID (uses randomblob(16) default)
INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com');
INSERT INTO users (name, email) VALUES ('Bob', 'bob@example.com');

-- Get the generated ID for foreign key relationships
INSERT INTO posts (author_id, title, content)
SELECT id, 'My First Post', 'Hello World!' FROM users WHERE email = 'alice@example.com';

Generating Primary Keys in Your App

For TEXT UUIDs:

import "github.com/google/uuid"

// Generate new UUID as string
userID := uuid.New().String()  // "550e8400-e29b-41d4-a716-446655440000"

// Insert into SQLite
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (?, ?, ?)",
    userID, "Alice", "alice@example.com")

For BLOB random bytes:

// Option A: Use SQLite's randomblob() function directly
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (randomblob(16), ?, ?)",
    "Alice", "alice@example.com")

// Option B: Generate random bytes in Go
import "crypto/rand"

randomID := make([]byte, 16)
rand.Read(randomID)
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (?, ?, ?)",
    randomID, "Alice", "alice@example.com")

// Option C: Use UUID as bytes (if you need UUID format)
import "github.com/google/uuid"

userID := uuid.New()
userIDBytes, _ := userID.MarshalBinary()  // Convert to []byte
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (?, ?, ?)",
    userIDBytes, "Alice", "alice@example.com")

It is up to you to decide which primary key format to use. TEXT UUIDs easier to work with, while BLOB UUIDs are a little more efficient in SQLite. Also technically speaking when you use BLOB primary keys with randomblob(), you are not using UUIDs at all. It’s just random bytes. But it provides enough uniqueness for virtually any use case. To keep documentation simple, we’ll refer to both as “UUIDs” even though technically BLOB with randomblob() are not UUIDs.

Why UUID Primary Keys?

  1. Global Uniqueness: UUIDs prevent conflicts when multiple devices create records offline
  2. Sync Compatibility: The server expects UUID format for proper change tracking
  3. Cross-Device Safety: No risk of primary key collisions between devices
  4. Idempotency: Enables safe retry of sync operations

Important: Do not use INTEGER PRIMARY KEY or auto-incrementing IDs with go-oversync. These will cause sync conflicts when multiple devices create records offline. Also as of now go-oversync expects UUID as the primary key format. This may change in the future.

Custom Primary Key Columns

By default, oversqlite assumes all tables use “id” as their primary key column. If your existing database schema uses different primary key column names, you can specify them:

// Example: Tables with custom primary key columns
cfg := oversqlite.DefaultConfig("business", []oversqlite.SyncTable{
    {TableName: "users", SyncKeyColumnName: "user_uuid"},       // Uses "user_uuid" as PK
    {TableName: "products", SyncKeyColumnName: "product_code"}, // Uses "product_code" as PK
    {TableName: "posts"}, // Empty string defaults to "id"
})

// Your SQLite table schema must look like:
// CREATE TABLE users (user_uuid TEXT PRIMARY KEY, name TEXT, email TEXT)
// CREATE TABLE products (product_code TEXT PRIMARY KEY, name TEXT, price REAL)
// CREATE TABLE posts (id TEXT PRIMARY KEY, title TEXT, content TEXT)