Getting Started
This guide walks you through creating your own simple HTTP server with go-oversync integration. You’ll build a complete sync-enabled server with users and posts tables from scratch.
Prerequisites
- PostgreSQL running locally (Docker example below)
- Go 1.21+
Core Concepts
Before building your sync server, it’s important to understand the key components that make go-oversync work:
User ID
- What it is: A unique identifier for each user in your system
- Purpose: Provides complete data isolation between users
- Example:
"user-jane","user-123", or any string that uniquely identifies a user - Source: Extracted from your authentication system (JWT
subclaim, session data, etc.)
Device ID (aka Source ID)
- What it is: A stable, unique identifier for each device/client installation
- Purpose: Critical for sync reliability, idempotency, and preventing infinite loops
- Example:
"device-123","phone-abc", or a UUID generated per app installation - Source: Generated by client app and stored locally, or extracted from JWT
didclaim
Why Device IDs are Essential:
- Idempotency Protection
- Prevents duplicate processing when network requests are retried
- Server uses (User ID + Device ID + Source Change ID) as unique key
- Same change uploaded multiple times is safely ignored
- Echo Suppression
- Devices don’t download their own changes back
- Prevents infinite sync loops where Device A uploads → Device A downloads same change
- Server excludes changes from requesting Device ID during downloads
- Conflict Resolution
- Helps identify which device made which changes
- Enables “last writer wins” or custom conflict resolution strategies
- Audit trail for debugging sync issues
- Hydration Recovery
- When app is reinstalled, new Device ID triggers full hydration
- Server sends all user’s data (include_self=true) for complete recovery
- Prevents partial sync state after fresh installation
Server Version
- What it is: An incrementing number for each row that changes on every update
- Purpose: Optimistic concurrency control - detects when two devices edit the same row
- Example: Row starts at version 0, becomes 1 after first edit, 2 after second edit, etc.
- Conflict: If device A tries to update version 1 but server has version 2, it’s a conflict
Server ID (Sequence)
- What it is: A global, incrementing sequence number for all changes across all users
- Purpose: Provides ordered change stream for downloads and pagination
- Example: Change 1, 2, 3, 4… across the entire system
- Usage: Clients download “all changes after sequence 100”
Source Change ID
- What it is: A per-device sequence number for changes from that device
- Purpose: Idempotency - prevents duplicate processing of the same change
- Example: Device sends changes 1, 2, 3… and server can detect/ignore duplicates
- Uniqueness: Combined with User ID + Device ID forms a unique key
Sync Tables
go-oversync creates these tables automatically to manage sync metadata:
sync.sync_row_meta— Tracks version and deleted status for each rowsync.sync_state— Stores current payload for each synced rowsync.server_change_log— Ordered log of all changes for download streams
Your business tables (like users and posts) remain completely unchanged. Well, almost
unchanged, we migrate foreign keys to deferred to support deferred batch ordering.
How Sync Works
Here’s how these components work together in a typical sync flow:
Upload Flow:
- Client makes changes to local SQLite database
- Client sends changes with: User ID, Device ID, Source Change ID, Server Version
- Server checks: “Is this User ID + Device ID + Source Change ID already processed?” (idempotency)
- Server checks: “Does the Server Version match current version?” (conflict detection)
- If valid: Server applies change, increments Server Version, assigns Server ID
- Server responds with new Server Version or conflict details
Download Flow:
- Client requests: “Give me changes after Server ID X for my User ID”
- Server returns: Ordered list of changes (excluding client’s own Device ID)
- Client applies changes locally and updates its “last seen Server ID”
- Process repeats to stay in sync
Multi-Device Example:
- Phone (device-phone) creates user → Server ID 1, Server Version 1
- Laptop (device-laptop) downloads → Gets the user creation
- Laptop edits user → Server ID 2, Server Version 2
- Phone downloads → Gets the user edit
- Both devices now have identical data
Step 1: Set Up PostgreSQL
Start PostgreSQL and create a database:
docker run --name oversync-pg -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:16
createdb my_sync_app # or use psql to create the DB
Step 2: Create Your Project
Initialize a new Go project:
mkdir my-sync-server
cd my-sync-server
go mod init my-sync-server
go get github.com/mobiletoly/go-oversync
go get github.com/jackc/pgx/v5/pgxpool
Step 3: Create Your Business Tables
Create a SQL file schema.sql to define your business tables:
-- Create business schema
CREATE SCHEMA IF NOT EXISTS business;
-- Users table
CREATE TABLE IF NOT EXISTS business.users (
id UUID PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Posts table with foreign key to users
CREATE TABLE IF NOT EXISTS business.posts (
id UUID PRIMARY KEY,
author_id UUID NOT NULL REFERENCES business.users(id) ON DELETE CASCADE,
title TEXT NOT NULL,
content TEXT,
created_at TIMESTAMPTZ DEFAULT NOW()
);
Step 4: Create Your Server
Create main.go with your HTTP server:
package main
import (
"context"
"database/sql"
"encoding/json"
"log"
"log/slog"
"net/http"
"os"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/mobiletoly/go-oversync/oversync"
_ "github.com/jackc/pgx/v5/stdlib"
)
func main() {
ctx := context.Background()
logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
// Connect to PostgreSQL
databaseURL := "postgres://postgres:postgres@localhost:5432/my_sync_app?sslmode=disable"
pool, err := pgxpool.New(ctx, databaseURL)
if err != nil {
log.Fatal("Failed to connect to database:", err)
}
defer pool.Close()
// Initialize business tables
if err := initializeBusinessTables(ctx, pool); err != nil {
log.Fatal("Failed to initialize business tables:", err)
}
// Configure go-oversync
cfg := &oversync.ServiceConfig{
MaxSupportedSchemaVersion: 1, // Version for client compatibility checks
AppName: "my-sync-app",
RegisteredTables: []oversync.RegisteredTable{
// Only register tables you want to sync
{Schema: "business", Table: "users"},
{Schema: "business", Table: "posts"},
},
}
// Create sync service
svc, err := oversync.NewSyncService(pool, cfg, logger)
if err != nil {
log.Fatal("Failed to create sync service:", err)
}
// Set up authentication (using JWT for this example)
// JWT will contain 'sub' (User ID) and 'did' (Device ID) claims
jwtSecret := "your-secret-key-change-in-production"
jwtAuth := oversync.NewJWTAuth(jwtSecret)
// Create sync handlers. Here we pass the JWT authenticator directly, but you can pass
// your own authenticator that implements the oversync.ClientAuthenticator interface
h := oversync.NewSyncHandlers(svc, jwtAuth, logger)
mux := http.NewServeMux()
// Authentication endpoint to return JWT token to the client for sync
mux.HandleFunc("POST /signin", func(w http.ResponseWriter, r *http.Request) {
handleSignIn(w, r, jwtAuth)
})
// If needed, add other authentication endpoints, e.g. to refresh tokens or to sign out
// Sync endpoints (protected by JWT authentication)
mux.Handle("POST /sync/upload", jwtAuth.Middleware(http.HandlerFunc(h.HandleUpload)))
mux.Handle("GET /sync/download", jwtAuth.Middleware(http.HandlerFunc(h.HandleDownload)))
logger.Info("Server starting on :8080")
if err := http.ListenAndServe(":8080", mux); err != nil {
log.Fatal("Server failed:", err)
}
}
func initializeBusinessTables(ctx context.Context, pool *pgxpool.Pool) error {
// Read and execute schema.sql
schema := `
CREATE SCHEMA IF NOT EXISTS business;
CREATE TABLE IF NOT EXISTS business.users (
id UUID PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS business.posts (
id UUID PRIMARY KEY,
author_id UUID NOT NULL REFERENCES business.users(id) ON DELETE CASCADE,
title TEXT NOT NULL,
content TEXT,
created_at TIMESTAMPTZ DEFAULT NOW()
);
`
_, err := pool.Exec(ctx, schema)
return err
}
type SignInRequest struct {
Username string `json:"username"`
Password string `json:"password"`
DeviceID string `json:"device_id"`
}
type SignInResponse struct {
Token string `json:"token"`
UserID string `json:"user_id"`
DeviceID string `json:"device_id"`
}
func handleSignIn(w http.ResponseWriter, r *http.Request, jwtAuth *oversync.JWTAuth) {
var req SignInRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "Invalid JSON", http.StatusBadRequest)
return
}
// Validate req.Username and req.Password, you can allow empty password for demo
// ...
// Generate User ID based on username (in production, this comes from your user database)
// This becomes the JWT 'sub' claim and provides complete data isolation per user
userID := "user-" + req.Username
// Device ID comes from client (usually generated once per app installation)
// This becomes the JWT 'did' claim and enables idempotency + echo suppression
deviceID := req.DeviceID
// Generate JWT token with User ID and Device ID (1 hour expiry)
// go-oversync will extract these from the JWT for all sync operations
token, err := jwtAuth.GenerateToken(userID, deviceID, 1*time.Hour)
if err != nil {
http.Error(w, "Failed to generate token", http.StatusInternalServerError)
return
}
// Return the token and IDs for client reference
resp := SignInResponse{
Token: token,
UserID: userID,
DeviceID: deviceID,
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
}
Note on authentication
For convenience, go-oversync provides a small helper package at
github.com/mobiletoly/go-oversync/oversync/jwtwith a ready-to-use JWT middleware and token utilities. It expectssub(user id) anddid(device id) claims and works out of the box with the sync handlers.You are not required to use this package: you can plug in your own authorization and identity extraction (sessions, custom JWTs, API keys, etc.). Just ensure your middleware sets user and device identity consistently so uploads/downloads are scoped correctly and idempotency works as designed.
Step 5: Run Your Server
Start your server:
go run main.go
Your server will:
- Connect to PostgreSQL and initialize business tables
- Set up go-oversync sidecar tables automatically
- Start listening on
:8080with sync endpoints ready
Step 6: Test Your Sync Server
Create a Test Client
Create a new file client_test.go to test your sync server using the oversqlite client:
package main
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"log"
"net/http"
"os"
"strings"
"github.com/google/uuid"
"github.com/mobiletoly/go-oversync/oversqlite"
_ "github.com/mattn/go-sqlite3"
)
type SignInRequest struct {
Username string `json:"username"`
Password string `json:"password"`
DeviceID string `json:"device_id"`
}
type SignInResponse struct {
Token string `json:"token"`
UserID string `json:"user_id"`
DeviceID string `json:"device_id"`
}
func main() {
ctx := context.Background()
// Create SQLite database
db, err := sql.Open("sqlite3", "test_client.db")
if err != nil {
log.Fatal("Failed to open SQLite database:", err)
}
defer db.Close()
// Create business tables in SQLite
if err := createClientTables(db); err != nil {
log.Fatal("Failed to create client tables:", err)
}
// Generate or retrieve Device ID (in production, store this persistently)
deviceID := getOrCreateDeviceID()
// Sign in to get token (contains User ID and Device ID)
userID, _, token, err := signIn("jane", "any-password", deviceID)
if err != nil {
log.Fatal("Failed to sign in:", err)
}
fmt.Printf("Signed in as %s with device %s\n", userID, deviceID)
// Configure oversqlite client for the tables we want to sync
cfg := oversqlite.DefaultConfig("business", []oversqlite.SyncTable{
{TableName: "users"},
{TableName: "posts"},
})
// Token provider function - oversqlite will call this to get auth tokens
tokenProvider := func(ctx context.Context) (string, error) {
return token, nil
}
// Create sync client with User ID and Device ID
// These must match what's in the JWT token for authentication to work
client, err := oversqlite.NewClient(
db, // Local SQLite database
"http://localhost:8080", // Your sync server URL
userID, // User ID for data isolation
deviceID, // Device ID for idempotency and echo suppression
tokenProvider, // Function to get auth tokens
cfg, // Table configuration
)
if err != nil {
log.Fatal("Failed to create sync client:", err)
}
// Test the sync flow
if err := testSyncFlow(ctx, db, client); err != nil {
log.Fatal("Sync test failed:", err)
}
fmt.Println("✅ Sync test completed successfully!")
}
Now add the helper functions to complete the client:
func signIn(username, password, deviceID string) (string, string, string, error) {
reqBody := SignInRequest{
Username: username,
Password: password,
DeviceID: deviceID,
}
jsonData, err := json.Marshal(reqBody)
if err != nil {
return "", "", "", err
}
resp, err := http.Post("http://localhost:8080/signin", "application/json", strings.NewReader(string(jsonData)))
if err != nil {
return "", "", "", err
}
defer resp.Body.Close()
var signInResp SignInResponse
if err := json.NewDecoder(resp.Body).Decode(&signInResp); err != nil {
return "", "", "", err
}
return signInResp.UserID, signInResp.DeviceID, signInResp.Token, nil
}
func createClientTables(db *sql.DB) error {
schema := `
CREATE TABLE IF NOT EXISTS users (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS posts (
id TEXT PRIMARY KEY,
author_id TEXT NOT NULL,
title TEXT NOT NULL,
content TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (author_id) REFERENCES users(id) ON DELETE CASCADE
);
`
_, err := db.Exec(schema)
return err
}
func testSyncFlow(ctx context.Context, db *sql.DB, client *oversqlite.Client) error {
// Step 1: Insert a user locally (normal SQLite INSERT)
// oversqlite automatically tracks this change via triggers
userID := uuid.New().String()
_, err := db.ExecContext(ctx, `
INSERT INTO users (id, name, email)
VALUES (?, ?, ?)
`, userID, "Jane Doe", "jane@example.com")
if err != nil {
return fmt.Errorf("failed to insert user: %w", err)
}
fmt.Println("Created user locally (Server Version: 0, Source Change ID: auto-assigned)")
// Step 2: Upload changes to server
// Client sends: User ID, Device ID, Source Change ID, Server Version 0
// Server responds with: new Server Version 1, Server ID assigned
if err := client.UploadOnce(ctx); err != nil {
return fmt.Errorf("failed to upload: %w", err)
}
fmt.Println("Uploaded user to server (Server assigned Server Version 1)")
// Step 3: Insert a post locally with foreign key to user
postID := uuid.New().String()
_, err = db.ExecContext(ctx, `
INSERT INTO posts (id, author_id, title, content)
VALUES (?, ?, ?, ?)
`, postID, userID, "My First Post", "Hello, sync world!")
if err != nil {
return fmt.Errorf("failed to insert post: %w", err)
}
fmt.Println("Created post locally (Server Version: 0, new Source Change ID)")
// Step 4: Upload the post
// Server validates foreign key exists, assigns new Server ID and Server Version
if err := client.UploadOnce(ctx); err != nil {
return fmt.Errorf("failed to upload post: %w", err)
}
fmt.Println("Uploaded post to server (Server assigned Server Version 1)")
// Step 5: Download changes (simulating another device)
// Requests: "Give me changes after my last seen Server ID"
// Server returns: Changes from other devices (excludes our Device ID)
downloaded, hasMore, err := client.DownloadOnce(ctx, 100)
if err != nil {
return fmt.Errorf("failed to download: %w", err)
}
fmt.Printf("Downloaded %d changes from other devices (has_more: %t)\n", downloaded, hasMore)
// Step 6: Verify data exists locally
var userCount, postCount int
db.QueryRowContext(ctx, "SELECT COUNT(*) FROM users").Scan(&userCount)
db.QueryRowContext(ctx, "SELECT COUNT(*) FROM posts").Scan(&postCount)
fmt.Printf("Local database: %d users, %d posts\n", userCount, postCount)
fmt.Println("All sync metadata is stored in separate _sync_* tables")
return nil
}
// getOrCreateDeviceID demonstrates Device ID best practices
// In production, store this in secure, persistent storage
func getOrCreateDeviceID() string {
// For this example, we'll use a simple file-based approach
// In mobile apps, use Keychain (iOS) or EncryptedSharedPreferences (Android)
const deviceIDFile = "device_id.txt"
// Try to read existing Device ID
if data, err := os.ReadFile(deviceIDFile); err == nil {
deviceID := strings.TrimSpace(string(data))
if deviceID != "" {
fmt.Printf("Using existing Device ID: %s\n", deviceID)
return deviceID
}
}
// Generate new Device ID
deviceID := "desktop-" + uuid.New().String()
// Store it persistently
if err := os.WriteFile(deviceIDFile, []byte(deviceID), 0600); err != nil {
log.Printf("Warning: Could not save Device ID: %v", err)
}
fmt.Printf("Generated new Device ID: %s\n", deviceID)
return deviceID
}
Run the Test Client
Make sure your server is running:
go run main.go
This demonstrates:
- Authentication — Getting a JWT token from your
/signinendpoint - Local data creation — Inserting records into SQLite
- Upload sync — Sending changes to PostgreSQL server
- Download sync — Receiving changes from other devices
- Automatic change tracking — oversqlite handles all sync metadata
SQLite Primary Key Requirements
For go-oversync to work with your SQLite database, all primary key columns must be TEXT (or BLOB) type containing UUID values. This is a fundamental requirement for the sync system to properly track and identify records across devices.
Supported Primary Key Formats
Option 1: TEXT with UUID strings
CREATE TABLE users (
id TEXT PRIMARY KEY, -- UUID as string: "550e8400-e29b-41d4-a716-446655440000"
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL
);
CREATE TABLE posts (
id TEXT PRIMARY KEY, -- UUID as string
author_id TEXT NOT NULL, -- Foreign key also as UUID string
title TEXT NOT NULL,
content TEXT,
FOREIGN KEY (author_id) REFERENCES users(id)
);
Option 2: BLOB with random bytes
CREATE TABLE users (
id BLOB PRIMARY KEY, -- 16-byte random BLOB
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL
) WITHOUT ROWID;
CREATE TABLE posts (
id BLOB PRIMARY KEY, -- 16-byte random BLOB
author_id BLOB NOT NULL, -- Foreign key as BLOB
title TEXT NOT NULL,
content TEXT,
FOREIGN KEY (author_id) REFERENCES users(id)
) WITHOUT ROWID;
-- Insert with SQLite's randomblob() function
INSERT INTO users (id, name, email) VALUES (randomblob(16), 'Alice', 'alice@example.com');
INSERT INTO users (id, name, email) VALUES (randomblob(16), 'Bob', 'bob@example.com');
Option 2b: BLOB with randomblob() as default value
CREATE TABLE users (
id BLOB PRIMARY KEY DEFAULT (randomblob(16)), -- Auto-generate random 16-byte BLOB
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL
) WITHOUT ROWID;
CREATE TABLE posts (
id BLOB PRIMARY KEY DEFAULT (randomblob(16)), -- Auto-generate random 16-byte BLOB
author_id BLOB NOT NULL, -- Foreign key as BLOB
title TEXT NOT NULL,
content TEXT,
FOREIGN KEY (author_id) REFERENCES users(id)
) WITHOUT ROWID;
-- Insert without specifying ID (uses randomblob(16) default)
INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com');
INSERT INTO users (name, email) VALUES ('Bob', 'bob@example.com');
-- Get the generated ID for foreign key relationships
INSERT INTO posts (author_id, title, content)
SELECT id, 'My First Post', 'Hello World!' FROM users WHERE email = 'alice@example.com';
Generating Primary Keys in Your App
For TEXT UUIDs:
import "github.com/google/uuid"
// Generate new UUID as string
userID := uuid.New().String() // "550e8400-e29b-41d4-a716-446655440000"
// Insert into SQLite
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (?, ?, ?)",
userID, "Alice", "alice@example.com")
For BLOB random bytes:
// Option A: Use SQLite's randomblob() function directly
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (randomblob(16), ?, ?)",
"Alice", "alice@example.com")
// Option B: Generate random bytes in Go
import "crypto/rand"
randomID := make([]byte, 16)
rand.Read(randomID)
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (?, ?, ?)",
randomID, "Alice", "alice@example.com")
// Option C: Use UUID as bytes (if you need UUID format)
import "github.com/google/uuid"
userID := uuid.New()
userIDBytes, _ := userID.MarshalBinary() // Convert to []byte
_, err := db.Exec("INSERT INTO users (id, name, email) VALUES (?, ?, ?)",
userIDBytes, "Alice", "alice@example.com")
It is up to you to decide which primary key format to use. TEXT UUIDs easier to work with, while BLOB UUIDs are a little more efficient in SQLite. Also technically speaking when you use BLOB primary keys with randomblob(), you are not using UUIDs at all. It’s just random bytes. But it provides enough uniqueness for virtually any use case. To keep documentation simple, we’ll refer to both as “UUIDs” even though technically BLOB with randomblob() are not UUIDs.
Why UUID Primary Keys?
- Global Uniqueness: UUIDs prevent conflicts when multiple devices create records offline
- Sync Compatibility: The server expects UUID format for proper change tracking
- Cross-Device Safety: No risk of primary key collisions between devices
- Idempotency: Enables safe retry of sync operations
Important: Do not use INTEGER PRIMARY KEY or auto-incrementing IDs with go-oversync. These will cause sync conflicts when multiple devices create records offline. Also as of now go-oversync expects UUID as the primary key format. This may change in the future.
Custom Primary Key Columns
By default, oversqlite assumes all tables use “id” as their primary key column. If your existing database schema uses different primary key column names, you can specify them:
// Example: Tables with custom primary key columns
cfg := oversqlite.DefaultConfig("business", []oversqlite.SyncTable{
{TableName: "users", SyncKeyColumnName: "user_uuid"}, // Uses "user_uuid" as PK
{TableName: "products", SyncKeyColumnName: "product_code"}, // Uses "product_code" as PK
{TableName: "posts"}, // Empty string defaults to "id"
})
// Your SQLite table schema must look like:
// CREATE TABLE users (user_uuid TEXT PRIMARY KEY, name TEXT, email TEXT)
// CREATE TABLE products (product_code TEXT PRIMARY KEY, name TEXT, price REAL)
// CREATE TABLE posts (id TEXT PRIMARY KEY, title TEXT, content TEXT)