TileDB-SOMA: [Bug/Question] `Experiment.obs.count` has long wall clock time - is this normal?

Describe the bug Getting the count of obs from an experiment takes a while for large datasets (~17 seconds here). Is this normal?

I ran the profiling script reported here https://github.com/single-cell-data/TileDB-SOMA/issues/1181 on a test script (attached below).

with soma.Experiment.open(<URI>) as exp:
    res = exp.obs.count

Output of profiler:

{
  "command": "python test_obs_count.py",
  "timestamp": 1714999582.266067,
  "stdout": "",
  "stderr": "Command being timed: \"python test_obs_count.py\"\nUser time (seconds): 22.08\nSystem time (seconds): 1.94\nPercent of CPU this job got: 141%\nElapsed (wall clock) time (seconds): 16.96\nAverage shared text size (kbytes): 0\nAverage unshared data size (kbytes): 0\nAverage stack size (kbytes): 0\nAverage total size (kbytes): 0\nMaximum resident set size (kbytes): 1906388\nAverage resident set size (kbytes): 0\nMajor (requiring I/O) page faults: 0\nMinor (reclaiming a frame) page faults: 489891\nVoluntary context switches: 3020\nInvoluntary context switches: 164481\nSwaps: 0\nFile system inputs: 0\nFile system outputs: 0\nSocket messages sent: 0\nSocket messages received: 0\nSignals delivered: 0\nPage size (bytes): 4096\nExit status: 0\n",
  "tiledb_stats": {},
  "somacore_version": "1.0.10",
  "tiledbsoma_version": "1.10.1",
  "host_context": {
    "uname": [
      "Linux",
      "ip-172-31-24-57",
      "6.5.0-1018-aws",
      "#18~22.04.1-Ubuntu SMP Fri Apr  5 17:44:33 UTC 2024",
      "x86_64"
    ],
    "total_virtual_mem_bytes": 33644634112,
    "total_physical_mem_bytes": 33644634112,
    "swap_mem_total_bytes": 0,
    "swap_mem_used_bytes": 0,
    "swap_mem_free_bytes": 0,
    "swap_mem_pct_bytes": 0,
    "swap_mem_sin_bytes": 0,
    "swap_mem_sout_bytes": 0,
    "cpu_count": 8,
    "python_version": [
      3,
      10,
      12,
      "final",
      0
    ],
    "get_git_revision_hash": "e52092ec96eeaf1b7633476da925f89e1b74cadb"
  },
  "user_time_sec": "22.08",
  "system_time_sec": "1.94",
  "pct_of_cpu": 141,
  "elapsed_time_sec": "16.96",
  "avg_shared_text_sz_kb": 0,
  "avg_unshared_text_sz_kb": 0,
  "avg_stack_sz_kb": 0,
  "avg_total_sz_kb": 0,
  "max_res_set_sz_kb": 1906388,
  "avg_res_set_sz_kb": 0,
  "major_page_faults": 0,
  "minor_page_faults": 489891,
  "voluntary_context_switches": 3020,
  "involuntary_context_switches": 164481,
  "swaps": 0,
  "file_system_inputs": 0,
  "file_system_outputs": 0,
  "socket_messages_sent": 0,
  "socket_messages_received": 0,
  "signals_delivered": 0,
  "page_size_bytes": 4096,
  "exit_status": 0,
  "custom_out": [
    null,
    null
  ],
  "command_key": "4e379762d2a949439b7ed5b3da01dd05"
}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (11 by maintainers)

Most upvoted comments

@mdylan2:

Thanks for the detailed gist! Using this, I’ve found:

For the moment, two workarounds:

  • Consolidating your two fragments of obs into one (array-fragment consolidation) will resolve this
  • A workaround: 1+exp.obs.non_empty_domain()[0][1]. Rationale:
    • SOMA does not assume all soma_joinid values are contiguous starting from zero
    • But in practice they usually are – the above will be correct, and quicker than exp.obs.count.

We’ll brainstorm a couple ideas to get to doing the right thing more often here.

Included in the gist

This is a local disk query.

The output is 24,448,693. Not setting soma.init_buffer_bytes so assuming it’s default here