Request for feedback: optional C2PA metadata extraction

Development
1

Hi everyone,

I’ve been working on an initial C2PA metadata extraction implementation for LocalSearch and would appreciate feedback on whether the concept makes sense for the project before opening a merge request.

C2PA is a Linux Foundation standard for recording the origin and editing history of digital media. Adopters include Adobe, Microsoft, Google, Meta, OpenAI, Sony, Leica, Nikon, and Truepic. Notably, images generated by major AI services such as OpenAI ChatGPT, Google Gemini, and Microsoft Bing Image Creator include C2PA records, as do photos taken with some devices, including the Google Pixel 10 and various cameras from Leica, Sony, Canon, and Fujifilm.

The current branch is here:

The implementation:

  • Adds an optional c2pa-c-ffi Meson dependency.
  • Uses the C2PA SDK’s C API to read manifests from files.
  • Adds C2PA support to JPEG, PNG, GIF, TIFF, WebP, MP3, and the generic libav audio/video extractor.
  • Projects selected metadata onto existing LocalSearch properties:
    • active manifest instance ID → dc:identifier
    • c2pa.created.when → nie:contentCreated
    • software agent → nie:generator
  • Preserves metadata from existing extractors when already available.
  • Includes unit tests and functional fixtures for the supported formats.
  • Leaves C2PA disabled automatically when the dependency is unavailable.

The main open issue is dependency distribution. The C2PA project currently publishes the native library and header, but not pkg-config metadata, and, as far as I know, c2pa-c-ffi is not currently packaged by any Linux distribution. For local testing I used the official Linux release and supplied a temporary .pc file outside the repository.

Some questions you might want to consider are:

  1. Whether using c2pa-c-ffi as an optional system dependency is the right integration model.
  2. Whether the selected mappings onto existing LocalSearch properties are appropriate.
  3. Whether the generic libav integration is preferable to adding format-specific hooks.
  4. Whether the dependency packaging issue should be resolved through distribution packaging or elsewhere before a LocalSearch merge request.
2

Hey Anders. Thanks for the work here! I’ll leave someone else to discuss the engineering side.

My main question is about the design side of things, what does this feature enable users of GNOME to do that they couldn’t do before?

3

Hi Sam. In my mind, the main new user capabilities are:

1. The ability to search for and filter media “self-identifying” as having been generated by C2PA-enabled software such as ChatGPT and Google Gemini.
For instance, to find images generated by recent versions of ChatGPT, the user should be able to do:

localsearch search --images gpt-image 2.0

or:

tinysparql query \
  -b org.freedesktop.LocalSearch3 \
  -q 'SELECT ?file ?generator WHERE {
         ?file a nfo:Image ;
               nie:generator "gpt-image/2.0" .
    }'

2. Media producers may record media creation timestamps in c2pa.created.when.
Where simple filesystem mtimes may change as the file is moved around, the timestamp recorded in c2pa.created.when travels with the media’s provenance metadata. My implementation surfaces that timestamp as nie:contentCreated. That helps searches for “that photo I took in June of last year” or “the image I generated with ChatGPT last week”, etc., regardless of how much the file has been moved around since then.

3. Potential future support for additional formats that are not currently covered by any extractors.
The C2PA SDK also supports formats such as AVIF, DNG, HEIF/HEIC, JPEG XL, and SVG, which LocalSearch does not currently extract metadata from. (Whether a given file in those formats contains C2PA metadata of course depends on the producer.)

4. Potential future support for richer provenance data.
C2PA manifests can describe not only the media resource represented by a file, but also the activities, software agents, and ingredients involved in creating it. With appropriate upstream support for an ontology that can represent these provenance relationships, LocalSearch could eventually represent the creation, editing, conversion, and watermarking activities, as well as relationships to explicitly identified input assets. This would make it possible for users to query provenance relationships between media resources to answer questions such as “what image was this video based on” or “what other images are derivatives of this image”, etc.

An additional interesting property of C2PA is that the metadata assertions are cryptographically signed, and my implementation rejects them if they fail cryptographic validation by the C2PA SDK. If an assertion is presented as being signed with Google’s private key, but the signature does not validate, then the assertion is not surfaced in LocalSearch. (However, note that it is of course trivial to use some other private key than Google’s to create a cryptographically valid but factually false assertion that, say, the content was generated by Google Gemini. That signer-identity and trust question is a different can of worms that I have chosen not to open in this initial implementation.)

4

Ok, thanks for the explanation. So at a high level this is for desktop users who are managing intentionally AI-generated content, is that right?

Certainly seems interesting to have this possibility. Have you thought about how this could be exposed in the desktop? (Presumably the Files search dropdown would be the place).

5

On a high level, that is indeed the first near-term use case I foresee.

C2PA is a versatile metadata format, however, and not inherently linked to AI. You can get a better sense of its breadth by looking at the examples here: Examples of Content Credentials | example-assets. Compare the JSON manifests (the raw representation returned by the low-level C2PA reader) with the user-friendly presentation in the linked Adobe Content Authenticity Inspect (ACA Inspect) page in the “Inspect Online” column of each asset. As you can see, Adobe Photoshop and Adobe Lightroom may also embed C2PA metadata. Similarly, though not listed among the examples on that page, media capture devices such as cameras may also do so: this is an example of a manifest embedded in a photo taken with my own Google Pixel phone.

An open challenge is how (or whether) to map such rich but heterogeneous metadata into the LocalSearch ontology. I picked AI image generator metadata for this first implementation because I could map it quite cleanly into the standard Nepomuk and Dublin Core properties nie:generator, nie:contentCreated and dc:identifier. Files generated by other producers may not map as cleanly: for instance the manifest from my Google Pixel photo linked above contains a free-text "Created by Pixel Camera." description at a particular node in the manifest. It was not obvious to me how to map that into the LocalSearch ontology.

One idea I’ve had is to expand the LocalSearch ontology to include something like PROV-O. C2PA and the PROV ontology at their core both deal with the concept of the “provenance” of things, and as such the PROV vocabulary should be well-suited for expressing at least some of the types of relationships between media assets that C2PA can express. I don’t know how other developers and maintainers of the project feel about that, though. Would people be on board with expanding the LocalSearch ontology beyond Nepomuk, Dublin Core etc.?

As for how to expose the modelled data in the desktop, I have to admit that I am no expert in the GNOME UI; but yes, the Files search dropdown likely would be a great place for it. If that is not wanted, alternatively the modelled data could just be available for more specialized third-party apps that use LocalSearch as a backend.

6

PROV vocabulary should be well-suited for expressing at least some of the types of relationships between media assets that C2PA can express. I don’t know how other developers and maintainers of the project feel about that, though. Would people be on board with expanding the LocalSearch ontology beyond Nepomuk, Dublin Core etc.?

It’s always been a goal to make this possible. Carlos put a lot of work into TinySPARQL over the last few years to make it fully SPARQL 1.1 compliant. However, it does not support OWL, and I don’t know of any plans to implement that. It would be rather a big job as, of course, TinySPARQL is really SQLite in a trenchcoat and not a completely generic RDF store.

I suspect the practical option would be to extend Nepomuk to cover what users need, rather than trying to use PROV directly. (The Nepomuk project is dead since a long time ago, and in practice Localsearch uses a private fork maintained in-tree, so extending it is not difficult).

7

I like that idea better. Mapping to an external ontology like PROV is best left as an exercise to those who need it.

I’ve submitted my merge request for the first phase here: extractor: Add optional C2PA metadata support (!698) · Merge requests · GNOME / LocalSearch · GitLab

Below I’ve sketched out the sort of terms I believe it might take to model C2PA provenance in full in a subsequent phase:

Resources

Term Meaning
:Manifest A provenance manifest associated with an information element.
:Claim A signed claim that organizes assertions in a manifest.
:Activity An event or operation that affects media or its provenance.
:Agent A person, organization, device, service, or software participant in provenance.
:Ingredient A qualified reference to media or another representation used as an input.
:Assertion A provenance assertion, preserving its source label and payload.
:Binding A cryptographic relationship between provenance data and content.
:DataBinding A binding over content data bytes.
:BoxBinding A binding over named boxes or byte ranges.
:SoftBinding A binding based on a soft, non-exact association with content.
:Signature A signature over a claim and its associated signing information.
:ValidationResult The result of validating a signature, assertion, credential, or binding.

Manifest and claim relationships

Term Meaning
:hasManifest Associates an information element with a provenance manifest.
:activeManifest Identifies the manifest currently selected as active for a resource or ingredient.
:hasClaim Associates a manifest with its claim.
:hasAssertion Associates a claim with an assertion.
:hasSignature Associates a claim with its signature.
:describesResource Associates a manifest with the information element it describes.
:claimGenerator Associates a claim with the agent that generated it.
:claimSchemaVersion Records the source claim layout or schema version.
:assertionReference Records a claim’s reference to an assertion.
:createdAssertion Identifies an assertion created by the claim generator.
:gatheredAssertion Identifies an assertion supplied by another workflow component.
:redactedAssertion Identifies an assertion omitted or redacted from a claim or lineage.
:certificateStatus Associates a claim with its certificate-status assertion.
:assertionGroup Records the source grouping of an assertion.
:assertionLabel Records the complete source label of an assertion.
:rawPayload Records an assertion’s serialized or embedded payload.

Activities and agents

Term Meaning
:hasActivity Associates a manifest with an activity.
:activityType The normalized action type of an activity.
:originalActionName The original action name supplied by the source.
:atTime The time associated with an activity.
:performedBy Associates an activity with the agent that performed it.
:hasParameter Associates an activity with one of its parameters.
:hasDigitalSourceType Associates an activity with a digital-source-type concept.
:actionAssertion Associates a normalized activity with its source action assertion.
:uses Associates an activity with an ingredient it uses.
:produces Associates an activity with the information element it produces.
:agentName The source name of an agent.
:agentVersion The source version of an agent.
:organization Associates an agent with an organization.
:agentRole The role an agent plays in a provenance relationship.
:claimGeneratorRole The role of an agent that generates a claim.
:softwareAgentRole The role of software or hardware that performs an activity.
:creatorRole The role of an agent that creates content.
:authorRole The role of an author or human contributor.
:signerRole The role of an agent that signs a claim.
:issuerRole The role of an agent that issues a signing certificate.
:deviceRole The role of a device involved in capture or processing.

Ingredients and derivation

Term Meaning
:hasIngredient Associates an activity with an ingredient it uses.
:referencesResource Associates an ingredient with the information element it represents.
:ingredientRelationship Records the relationship of an ingredient to the current work.
:binding Associates a resource with a cryptographic binding.
:thumbnail Associates a resource with a thumbnail assertion.
:icon Associates a resource or agent with an icon assertion.
:derivedFrom Associates an information element with an input or prior representation.
:precedes Orders one activity before another activity.

Binding details

Term Meaning
:hashAlgorithm The algorithm used to calculate a binding digest.
:hashValue The digest value of a binding.
:bindingScope The content coverage described by a binding.
:includedRange A byte or box range included in a binding.
:excludedRange A byte or box range excluded from a binding.
:boundAssertion Associates a binding with an assertion it covers.
:boundResource Associates a binding with a resource it covers.

Signatures and validation

Term Meaning
:signatureAlgorithm The algorithm used to create a signature.
:signatureValue The signature bytes or serialized signature value.
:signedClaim Associates a signature with the claim it covers.
:signingAgent Associates a signature with the agent that signed the claim.
:certificateIssuer The issuer of the signing certificate.
:certificateName The subject or display name of the signing certificate.
:signatureTime The time associated with a signature.
:hasValidation Associates a signature with validation results.
:validationStatus The success, failure, or informational status of validation.
:validationCode A machine-readable validation result code.
:validationTarget The assertion, claim, resource, or URI to which a validation result applies.
:validationMessage A human-readable explanation of a validation result.
:validationTime The time at which validation occurred.
:credentialTrust The trust state assigned to a credential.

Activity values

The following values identify normalized activity types:

:created
:opened
:edited
:placed
:converted
:transcoded
:watermarked
:watermarkedUnbound
8

Thank you for your work here @afeder! IMO this sounds like worthwhile information to extract from files, I’m adding your merge request to my review list.

It’s definitely feasible to extend the existing ontology to handle this information, and TinySparql will transparently convert ontology updates into database structure changes. I would suggest an entire new namespace for this information, since the semanticdesktop.org ontology that ours branched off is still nominally up, I would suggest picking something that doesn’t clash with it. tpo: or lpo: perhaps?, from TinySparql/LocalSearch Provenance Ontology?

9

Drifting offtopic, but given the time and motivation, I’d nowadays probably look first in implementing support for SHACL. When I read the spec, I had much higher hopes of a plausible implementation with fewer and more predictable performance sinks, compared to the multiple times I’ve gone through the flavors of OWL.