CIL harvesting and metadata transformation processes

[arwenhutt]

Descriptive summary

Write scripts for regular harvesting and transformation of content (files and metadata) from the Cell Image Library GitHub repository: https://github.com/slash-segmentation/CIL_Public_Data_JSON

Documentation

1. Outline of Ongoing Harvesting processes https://docs.google.com/document/d/1Eg2024XATxuwdzFtoLTNujalK2SDR_s6-Sg6zyUC9wE/edit#heading=h.caf5ayev1pza
2. Mapping for creation of subject heading ingest file and object level record ingest file: https://docs.google.com/spreadsheets/d/1NnYw3bgNraJ9hZCH1SOKijh_JvJh3So1LpDC24vu7h8/edit#gid=1321122337

Related work

• Create CSV export of CIL JSON data damspas#449
• R&D Test Queries against CIL endpoints damspas#436

[lsitu]

@arwenhutt Can you give me the permission to the share doc Mapping for creation of subject heading ingest file and object level record ingest file so that I can take a look? Thanks.

[arwenhutt]

@lsitu done!

[lsitu]

Thanks @arwenhutt. I am able to see it now.

[lsitu]

@arwenhutt Could you show me an example for row#75 Create a new row for child component for json file in Mapping for creation of subject heading ingest file and object level record ingest file? Thanks.

[arwenhutt]

@lsitu In other words, we want to add a final child component to the object with the source metadata (the json file itself) attached as a file. We aren't mapping all of the source metadata into the dams so this gives a user access to the full source metadata in case they need it.

Let me know if you would still like an example ingest file, I can do that but this week is very busy and I'm not sure when I'll be able to get to it!

[lsitu]

@arwenhutt I see. Thanks.

[lsitu]

@arwenhutt Are there any identifiers in the CIL object that can be used to check whether a JSON file (object) is ingested or not?
For date:creation mapping (row#7-9) in [Mapping for creation of subject heading ingest file and object level record ingest file],(https://docs.google.com/spreadsheets/d/1NnYw3bgNraJ9hZCH1SOKijh_JvJh3So1LpDC24vu7h8/edit#gid=1321122337), is three CIL_CCDB.CIL.CORE.ATTRIBUTION.DATE values in an array or just one? Could you show me an example? Thanks.

[mcritchlow]

Hi @lsitu - I can help with the 2nd part, I'm not clear on the 1st part of the question re: identifiers in the object.

For the dates, I believe it can be an array with 1-m values.

I did a quick grep in the github repo grep -iRl "DATE" ./ to find some examples.

Here's one: Version8_6/DATA/CIL_PUBLIC_DATA/40685.json

I'm not sure that answers your question or not.