Writing a pandas Sidecar for Tauri

I'm currently working on a Tauri project that needs to utilize a series of pandas scripts for data cleaning before loading them into my database in Tauri. While Tauri supports embedding external binaries the documentation is pretty limited and generic, so I hit a few problems that were Python specific, which I'm documenting here for future me.

Hello World

A quick search for "python sidecar" turned up the example-tauri-v2-python-server-sidecar repo, which worked for me out of the box. It also put me on the path to use PyInstaller for my own sidecar implementation.

After successfully building the app, I went ahead and stripped out as much of the code as I could to just run a basic hello program (since the example repo included complications like writing concurrent Rust 😅).

main.rs

// Prevents additional console window on Windows in release, DO NOT REMOVE!!
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
 
use tauri::Emitter;
use tauri_plugin_shell::process::{CommandChild, CommandEvent};
use tauri_plugin_shell::ShellExt;
use tauri::{App, Manager};
 
#[tokio::main]
async fn main() {
    // instantiate app
    let app = tauri::Builder::default()
        .plugin(tauri_plugin_dialog::init())
        .plugin(tauri_plugin_shell::init())
        .invoke_handler(tauri::generate_handler![])
        .build(tauri::generate_context!())
        .expect("error building the app");

    // run sidecar
    let app_handle = app.handle().clone();
    println!("[tauri] Creating sidecar...");
    let _status = run_sidecar(app_handle);
    println!("[tauri] Sidecar run");

    app.run(|_, _| {});
}

pub fn run_sidecar(app_handle: tauri::AppHandle) -> Result<(), String> {
    // Spawn sidecar
    let sidecar_command = app_handle.shell().sidecar("hello").unwrap();
    let (mut rx, _child) = sidecar_command.spawn().expect("Failed to spawn sidecar");

    // Spawn an async task to handle sidecar communication
    tauri::async_runtime::spawn(async move {
        while let Some(event) = rx.recv().await {
            match event {
                CommandEvent::Stdout(line_bytes) => {
                    let line = String::from_utf8_lossy(&line_bytes);
                    println!("Sidecar stdout: {}", line);
                    // Emit the line to the frontend
                    app_handle
                        .emit("sidecar-stdout", line.to_string())
                        .expect("Failed to emit sidecar stdout event");
                }
                CommandEvent::Stderr(line_bytes) => {
                    let line = String::from_utf8_lossy(&line_bytes);
                    eprintln!("Sidecar stderr: {}", line);
                    // Emit the error line to the frontend
                    app_handle
                        .emit("sidecar-stderr", line.to_string())
                        .expect("Failed to emit sidecar stderr event");
                }
                _ => {}
            }
        }
    });

    Ok(())
}

For the python compilation, all I needed to do was run:

pyinstaller -n hello-aarch64-apple-darwin hello.py

and make sure the binary was accessible at:

~/workspaces/my-app/src-tauri/binaries/hello/dist/hello-aarch64-apple-darwin

which produced the following in my terminal when run:

[tauri] Creating sidecar...
[tauri] Sidecar run
Sidecar stdout: hello world

My Complications

The actual sidecar I need to run included the following complications:

  1. Use additional reference files (xlsx, yaml)
  2. Use external libraries (pandas, etc)
  3. Ingest a variable number of arguments to the sidecar

Following the same steps as for my hello world example resulted in a binary that worked when called on its own, but when called within Tauri, I got the following error:

Sidecar stderr: [PYI-39539:ERROR] Failed to load Python shared library '/Users/mclare/workspaces/my-app/src-tauri/target/debug/_internal/Python': dlopen: dlopen(/Users/maryannewachter/workspaces/blt-app/src-tauri/target/debug/_internal/Python, 0x000A): tried: '/Users/mclare/workspaces/my-app/src-tauri/target/debug/_internal/Python' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/mclare/workspaces/my-app/src-tauri/target/debug/_internal/Python' (no such file), '/Users/mclare/workspaces/my-app/src-tauri/target/debug/_internal/Python' (no such file)

As it turned out, I needed to make three key changes to the pyinstaller command:

pyinstaller  --add-data references:references --onefile --name run-$(rustc -Vv | grep "host:" | cut -d ' ' -f 2) run.py

I needed to add a mapping for the external assets (csvs) referenced by the binary via the --add-data source:target flag

I also needed to indicate that the binary should be compiled as a single file (which is what produced the error above)

Finally, so I can automate this elsewhere, I needed to capture the results of the rustc -Vv into the resulting binary name, as required by Tauri

I set up my sidecar to accept a variable number of filepaths for input csvs to be batch processed, with the first argument as the output file location. That meant I needed to modify the file_names Vec passed from the frontend and prepend it with Tauri's app_data_dir

After removing the old sidecar, I was left with:

import.rs

#[tauri::command]
pub async fn process_csvs(
    app_handle: tauri::AppHandle,
    state: tauri::State<'_, AppState>,
    mut file_names: Vec<String>,
) -> Result<(), String> {
    let app_data_dir = app_handle
        .path()
        .app_data_dir()
        .map_err(|e| format!("Failed to get app data directory: {}", e))?
        .to_str()
        .ok_or_else(|| "Failed to convert path to string".to_string())?
        .to_string();
    file_names.insert(0, app_data_dir);
    let sidecar_command = app_handle
        .shell()
        .sidecar("run")
        .map_err(|e| format!("Failed to create sidecar command: {}", e))?
        .args(file_names);

    let (mut rx, _child) = sidecar_command
        .spawn()
        .map_err(|e| format!("Failed to spawn sidecar process: {}", e))?;

    tauri::async_runtime::spawn(async move {
        while let Some(event) = rx.recv().await {
            match event {
                CommandEvent::Stdout(line_bytes) => {
                    let line = String::from_utf8_lossy(&line_bytes);
                    println!("Sidecar stdout: {}", line);
                    // Emit the line to the frontend
                    app_handle
                        .emit("sidecar-stdout", line.to_string())
                        .expect("Failed to emit sidecar stdout event");
                }
                CommandEvent::Stderr(line_bytes) => {
                    let line = String::from_utf8_lossy(&line_bytes);
                    eprintln!("Sidecar stderr: {}", line);
                    // Emit the error line to the frontend
                    app_handle
                        .emit("sidecar-stderr", line.to_string())
                        .expect("Failed to emit sidecar stderr event");
                }
                _ => {}
            }
        }
    });
    Ok(())
}

Next Steps

I'm writing a temporary file to the app data directory, which I will then load into the database I'm building, but eventually, I could probably just stream the csv results directly to Rust. Ultimately, I should probably get rid of the Python dependency and port all the data frame manipulation to Polars, but this was a worth sidequest (ha!) to learn a bit more about Tauri's features.