Data Management Client Command Line Tool data_management_client_command_line_tool

DiCOS Data Management Client Command Line Tool (ddmc)

Advance Preparation

1. Before using DiCOS, you should have the following things :

   • A grid user certificate, tutorial

   • Join a VO (Virtual Organization), tutorial

   • A DiCOS account, tutorial

2. SSH log in DiCOS web UI with your DiCOS account:

   • dicos-ui.grid.sinica.edu.tw or

   • dicos-ui.twgrid.org

Commands

• Set up

  Initialize the voms proxy before using ddmc.

  voms-proxy-init -voms <VO>

  $ voms-proxy-init -voms twgrid
  

  • note:

    • <VO> is one the Virtual Organizations you are joining, for example : twgrid, atlas, ams02.cern.ch

    • If you have already initialized a voms proxy in last 12 hours (so the proxy is not expired), you don't need to do this step again. In fact, "ddmc" will remind you if you don't have a valid and unexpired voms proxy initialized.

  <hr/>

• Ping

  Connect to DiCOs Data Management (DDM) server to see if it's alive. It will return OK if everything is fine.

  ddmc ping

  $ ddmc ping
  RUCIO Version: 0.2.6
  DDM Ping OK!
  

  <hr/>

• Upload

  Upload and register your local file to a DDM dataset.

  ddmc ul|upload <local file path> <DDM dataset>

  $ ddmc ul /home/wchang/wjtest/wjtest17 wjtest_dataset1
  
  $ ddmc ul /home/wchang/wjtest/wjtest* wjtest_dataset1
  

  • note:

    • The <local file path> can be either absolute path or relative path.
    • Wildcard * is available in <local file path> to upload multiple files.
    • <local file path> has to be regular file(s); any directory (and everything under it) or irregular file will NOT be uploaded.
    • The <DDM dataset> MUST be defined in this command.
    • If the dataset <DDM dataset> has not existed in your scope yet, it will be automatically created first.

  <hr/>

• Download

  Download a file or all files in a container/dataset on DDM to your local directory.

  ddmc dl|download <DDM container/dataset/file> [<local directory path>]

  $ ddmc dl tid9907_1473730551_wj_test_pilotlog.txt /home/wchang
  

  • note:

    • When downloading a whole dataset or container, all files in it will be downloaded to the local directory WITHOUT any nested sub-directory.
    • <local directory path> MUST be a directory, not a file. If this argument is missed in the command, present working directory (pwd) will be the default directory.
    • Wildcard * is NOT supported in <DDM container/dataset/file>

  <hr/>

• Query

  List content in a container/dataset in your scope on DDM.

  ddmc ls [-r] [<DDM container/dataset>]

  $ ddmc ls wjtest_2015031701
   |    |- twgrid-user-wchang:wjtest09 [FILE]
   |    |- twgrid-user-wchang:wjtest10 [FILE]
   |    |- twgrid-user-wchang:wjtest11 [FILE]
   |    |- twgrid-user-wchang:wjtest12 [FILE]
   |    |- twgrid-user-wchang:wjtest13 [FILE]
   |    |- twgrid-user-wchang:wjtest14 [FILE]
   |    |- twgrid-user-wchang:wjtest15 [FILE]
   |    |- twgrid-user-wchang:wjtest16 [FILE]
   |    |- twgrid-user-wchang:wjtest18 [FILE]
  
  $ ddmc ls 
  twgrid-user-wchang:JobOutput [CONTAINER]
  twgrid-user-wchang:wjtest2194 [FILE]
  twgrid-user-wchang:wjtest_20150313 [DATASET]
  twgrid-user-wchang:wjtest_20150315 [DATASET]
  twgrid-user-wchang:wjtest_20150316 [DATASET]
  twgrid-user-wchang:wjtest_2015031601 [DATASET]
  twgrid-user-wchang:wjtest_2015031701 [DATASET]
  twgrid-user-wchang:wjtest_20150324 [DATASET]
  
  $ ddmc ls -r
  twgrid-user-wchang:JobOutput [CONTAINER]
   |    |- twgrid-user-wchang:wjtest_20150312 [DATASET]
   |    |    |- twgrid-user-wchang:wjtest111 [FILE]
   |    |- twgrid-user-wchang:wjtest_20150313 [DATASET]
   |    |    |- twgrid-user-wchang:wjtest112 [FILE]
   |    |    |- twgrid-user-wchang:wjtest113 [FILE]
  twgrid-user-wchang:wjtest2194 [FILE]
  twgrid-user-wchang:wjtest_20150313 [DATASET]
   |    |- twgrid-user-wchang:wjtest101 [FILE]
   |    |- twgrid-user-wchang:wjtest102 [FILE]
   |    |- twgrid-user-wchang:wjtest103 [FILE]
  ......
  

  • note:

    • If <DDM container/dataset> is missing in the command, content in the top scope will be listed.
    • With option '-r', one can list all files exhaustively (recursively) in the container/dataset.

  <hr/>

  List file information such as file size, checksum, and direct link.

  ddmc getlink <DDM file>

  $ ddmc getlink tid9885_1473730529_wj_test_pilotlog.txt
  

Note:

This command-line tool is meant to deal with massive local file upload/download only. To use complete data management functions (such as copy/paste, creating a new container/dataset, etc.), please go to DiCOS Data Management Webpage.