{"@context":"http://iiif.io/api/presentation/3/context.json","id":"https://weareavp.aviaryplatform.com/iiif/t727942127/manifest","type":"Manifest","label":{"en":["Finding the Right AI Tools for DAM: AVP’s Human-Centered Evaluation Framework- Henry Stewart DAM Webinars"]},"logo":"https://d9jk7wjtjpu5g.cloudfront.net/organizations/logo_images/000/000/007/original/avp-logo-login_2x.png?1692278773","metadata":[{"label":{"en":["Date"]},"value":{"en":["2022-03-15 (created)"]}},{"label":{"en":["Type"]},"value":{"en":["Webinar"]}},{"label":{"en":["Language"]},"value":{"en":["English (primary)"]}},{"label":{"en":["Description"]},"value":{"en":["\u003cp\u003eFinding the Right AI Tools for DAM: AVP’s Human-Centered Evaluation Framework\u003c/p\u003e\r\n\u003cp\u003e \u003c/p\u003e\r\n\u003cp\u003eAI holds the promise of enriching your DAM program, but with so many options in the market, how do you find, test, and integrate the right tools and services for your needs? In this webinar, AVP will share our AI evaluation framework for testing commercial and open-source AI tools that meet your organization's unique needs. Using our human-centered approach, we will equip you with the ability to:\u003c/p\u003e\r\n\r\n\r\nUnderstand your users and use cases for AI-enriched metadata\r\n\r\n\r\n\r\n\r\nIdentify and prioritize areas where AI can help most\r\n\r\n\r\n\r\n\r\nSelect candidate technologies based on use cases and priorities\r\n\r\n\r\n\r\n\r\nDevelop a strategy for quantitative and qualitative evaluation of technologies tailored to your needs\r\n\r\n\r\n\r\n\r\nEvaluate technologies through the lens of functional fit and bias risk\r\n\r\n\r\n\u003cp\u003e \u003c/p\u003e\r\n\u003cp\u003eSpeakers:\u003c/p\u003e\r\n\u003cp\u003eShawn Averkamp, AVP Senior Consultant\u003c/p\u003e\r\n\u003cp\u003eShawn specializes in bringing data innovation into production, helping clients explore creative, user-focused solutions to a wide range of data, strategy, and software development challenges, from research and development of responsible AI in libraries and archives to supporting supply chain analysis of cattle transactions and deforestation in Brazil. \u003c/p\u003e\r\n\u003cp\u003e\u003cbr\u003eJason Ulsh, AVP Consultant\u003c/p\u003e\r\n\u003cp\u003eJason supports clients with system selection, workflow analysis, and digital asset management needs. He has more than 10 years of experience using cognitive ethnography and user-centered design techniques to assist organizations with their data and asset management challenges.\u003c/p\u003e\r\n\u003cp\u003e\u003cbr\u003eKara Van Malssen, AVP Partner and Managing Director for Consulting\u003c/p\u003e\r\n\u003cp\u003eKara works with clients to bridge the technical, human, and business aspects of projects. Kara has supported numerous organizations with DAM planning and roadmapping, technology selection, system implementation and optimization, and user experience design.\u003c/p\u003e (summary)"]}},{"label":{"en":["Keyword"]},"value":{"en":["DAM","Digital Asset Management","AI","Henry Stewart"]}},{"label":{"en":["Duration"]},"value":{"en":["01:07:02"]}}],"summary":{"en":["\u003cp\u003eFinding the Right AI Tools for DAM: AVP\u0026rsquo;s Human-Centered Evaluation Framework\u003c/p\u003e\r\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\r\n\u003cp\u003eAI holds the promise of enriching your DAM program, but with so many options in the market, how do you find, test, and integrate the right tools and services for your needs? In this webinar, AVP will share our AI evaluation framework for testing commercial and open-source AI tools that meet your organization's unique needs. Using our human-centered approach, we will equip you with the ability to:\u003c/p\u003e\r\n\u003cul\u003e\r\n\u003cul\u003e\r\n\u003cli\u003eUnderstand your users and use cases for AI-enriched metadata\u003c/li\u003e\r\n\u003c/ul\u003e\r\n\u003c/ul\u003e\r\n\u003cul\u003e\r\n\u003cul\u003e\r\n\u003cli\u003eIdentify and prioritize areas where AI can help most\u003c/li\u003e\r\n\u003c/ul\u003e\r\n\u003c/ul\u003e\r\n\u003cul\u003e\r\n\u003cul\u003e\r\n\u003cli\u003eSelect candidate technologies based on use cases and priorities\u003c/li\u003e\r\n\u003c/ul\u003e\r\n\u003c/ul\u003e\r\n\u003cul\u003e\r\n\u003cul\u003e\r\n\u003cli\u003eDevelop a strategy for quantitative and qualitative evaluation of technologies tailored to your needs\u003c/li\u003e\r\n\u003c/ul\u003e\r\n\u003c/ul\u003e\r\n\u003cul\u003e\r\n\u003cul\u003e\r\n\u003cli\u003eEvaluate technologies through the lens of functional fit and bias risk\u003c/li\u003e\r\n\u003c/ul\u003e\r\n\u003c/ul\u003e\r\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\r\n\u003cp\u003eSpeakers:\u003c/p\u003e\r\n\u003cp\u003eShawn Averkamp, AVP Senior Consultant\u003c/p\u003e\r\n\u003cp\u003eShawn specializes in bringing data innovation into production, helping clients explore creative, user-focused solutions to a wide range of data, strategy, and software development challenges, from research and development of responsible AI in libraries and archives to supporting supply chain analysis of cattle transactions and deforestation in Brazil.\u0026nbsp;\u003c/p\u003e\r\n\u003cp\u003e\u003cbr /\u003eJason Ulsh, AVP Consultant\u003c/p\u003e\r\n\u003cp\u003eJason supports clients with system selection, workflow analysis, and digital asset management needs. He has more than 10 years of experience using cognitive ethnography and user-centered design techniques to assist organizations with their data and asset management challenges.\u003c/p\u003e\r\n\u003cp\u003e\u003cbr /\u003eKara Van Malssen, AVP Partner and Managing Director for Consulting\u003c/p\u003e\r\n\u003cp\u003eKara works with clients to bridge the technical, human, and business aspects of projects. Kara has supported numerous organizations with DAM planning and roadmapping, technology selection, system implementation and optimization, and user experience design.\u003c/p\u003e"]},"provider":[{"id":"https://weareavp.aviaryplatform.com/aboutus","type":"Agent","label":{"en":["AVP"]},"homepage":[{"id":"https://weareavp.aviaryplatform.com/","type":"Text","label":{"en":["AVP"]},"format":"text/html"}],"logo":[{"id":"https://d9jk7wjtjpu5g.cloudfront.net/organizations/logo_images/000/000/007/original/avp-logo-login_2x.png?1692278773","type":"Image"}]}],"thumbnail":[{"id":"https://d9jk7wjtjpu5g.cloudfront.net/collection_resource_files/thumbnails/000/154/647/small/FindingtheRightAIToolsforDAMAVP%E2%80%99sHumanCenteredEvaluationFramework.mp4_1648062031.jpg?1648047635","type":"Image","format":"image/jpeg"}],"items":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647","type":"Canvas","label":{"en":["Media File 1 of 2 - Finding_the_Right_AI_Tools_for_DAM_AVPs_Human_Centered_Evaluation_Framework20240423-104915-i5swhn.mp4"]},"duration":3222.50667,"width":640,"height":360,"thumbnail":[{"id":"https://d9jk7wjtjpu5g.cloudfront.net/collection_resource_files/thumbnails/000/154/647/small/FindingtheRightAIToolsforDAMAVP%E2%80%99sHumanCenteredEvaluationFramework.mp4_1648062031.jpg?1648047635","type":"Image","format":"image/jpeg"}],"items":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/content/1","type":"AnnotationPage","items":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/content/1/annotation/1","type":"Annotation","motivation":"painting","body":{"id":"https://aviary-p-weareavp.s3.wasabisys.com/collection_resource_files/resource_files/000/154/647/original/Finding_the_Right_AI_Tools_for_DAM_AVPs_Human_Centered_Evaluation_Framework20240423-104915-i5swhn.mp4?1713862836","type":"Video","format":"video/mp4","duration":3222.50667,"width":640,"height":360},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647","metadata":[]}]}],"annotations":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948","type":"AnnotationPage","label":{"en":["Transcript AVP Finding the Right AI Tools for DAM [Transcript]"]},"items":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/1","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Hello, everyone. I'm Jeanette Camping from Henry Stewart events. I'm delighted to welcome you to our latest webinar, \"Finding the Right AI Tools for DAM: AVP's Human-Centered Evaluation Framework.\" It's great to see such a global audience of DAM professionals registered today. We'll be having Q\u0026A at the end of the presentations if you send in your questions on your \"Goto Webinar panel and we're recording today's session and you'll all be sent a link to the recording later. I'm delighted to welcome our speakers from AVP. Thank you, Kara, over to you.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3.0,41.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/2","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen Thanks, Jeanette, and welcome everyone. Thank you so much for joining us today. We're really excited to share our people centered approach to evaluating and selecting AI tools for DAM with you today. Let's start with some introductions. Shawn, over to you.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=41.0,56.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/3","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Hi, I'm Shawn Averkamp. I'm a senior consultant with AVP.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=56.0,61.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/4","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen And Jason,","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=61.0,63.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/5","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh I'm Jason Ulsh a consultant at AVP.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=63.0,66.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/6","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen Thank you both. And I'm Kara Van Malssen and I'm the managing director for consulting at AVP, so I help shape the projects we work on with some of our amazing clients. Let me quickly walk you through the agenda where we get started. So in today's session, we are going to be walking you through an approach to evaluating and selecting A.I. tools that lean on the methods and mindsets of human centered design. Now, some of you might be familiar with this concept. We obviously didn't invent it, but it's something we rely on for everything we do.  At AVP, we connect humans and data, and we work hard to create solutions for digital asset management designed for use. So the human centered design framework helps us ensure we're building solutions that will really work for people. So our hope is by the end of this webinar that you will understand the ways that AI might be able to support your work and that you'll be able to identify opportunities to use AI to enrich your assets value, that you'll be able to apply this human centered approach to selecting and evaluating candidate tools, and that you'll feel more confident putting AI tools to practical use in a production environment. So, Jeanette, let's stop the slides for just a minute. And I'm going to go to Shawn for just a sec. I wonder if you could give us some background. How did we develop this approach that we're going to be sharing today? I think you're muted, Shawn . Thank you.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=66.0,159.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/7","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Yes. So in the past three years, we have to work with clients on big research and development projects that we're exploring different aspects of AI and digital assets. First one was the audiovisual metadata platform AMP or the AMMPD project from Indiana University, which we're still supporting. It's in its third grant funded round of development right now. And when it's finished, AMP will be an open source platform that will allow collection managers and digital asset managers to upload audio and video files and then create pipelines of AI tools using speech to text, video, OCR named entity recognition, and lots more. So from this experience, we learned a lot about evaluating commercial and open-source AI tools. We had to recommend different tools to try for the pilot project. And so we needed to come up with a way to evaluate all of these different tools. So we came up with this evaluation criteria, which you were going to see in the presentation today, and that really helped us start to put a frame around how to assess and evaluate tools for specific use cases. The other big project that we worked on was with the Library of Congress Labs, a project called Humans in the Loop, where we explored with them how organizations could use crowdsourcing to create training data for machine learning tasks. So again, that was more about training machine learning, which we're not going to get into today. But some of the things we learned from that project were how to take ethical approaches to assessing projects, doing selection of collections and just making sure that users and the subjects of collections are not harmed or misrepresented. So we did a lot of workshops and interviews with the Library of Congress staff on coming up with ways to make sure that the crowdsourcing and AI projects were engaging and ethical, but also that tasks and outputs were useful to end users of the outputs and the volunteers that were creating the training data. So from both of these projects, we learned a lot about AI tools in general for DAMS, but we also learned how to bring users into this assessment and how to incorporate that into the human centered design processes here at AVP, which is what we're going to talk about today.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=159.0,312.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/8","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen Thanks, Shawn. And Jason, you've been with us for about a year now. But you come from a corporate DAM background. Did you ever do any kind of AI evaluation in your previous role?","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=312.0,326.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/9","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh Yeah. At the last place, we were having trouble keeping up with our backlog, and we realized that the thing that was taking a lot of our time was just getting good enough metadata on photos, things like that. Our DAMS vendor had just enabled AI tools that we could play around with. We're like, well, let's set up a demo. And they had four different AI tagging services that we could use, and they were all geared towards different kinds of things. So we ran some assets through, got really weird results on some things and nothing on some other things. And we realized we were going to have to have some way to test this out. So we went around the internet. We asked our friends, our colleagues, and there really wasn't any kind of like framework or anything for testing this stuff. So we just invented one. We made a spreadsheet and we like did very subjective evaluation of like are these tags good? Do we agree with this? Do we not agree with this? And we couldn't come up with anything that was really like that. We would feel confident going to somebody who could purchase something and saying, like, Yeah, this is the one we want, so we just abandon the whole thing. We would have loved to have a tool like what we're going to show you today to help us with this because it just didn't exist at the time we were doing. So hopefully the thing we show you today will allow you to make like you and the audience to make a good decision, that you feel confident.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=326.0,407.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/10","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen Thanks, Jason, for sharing that experience, and it's great to have you on our team. So just so you know, we can actually help organizations like the one Jason used to work for, for example, with that sort of evaluation get you to an actionable outcome so you can feel confident and have a clear understanding of the impact of AI like Shawn said, understanding all the implications, including the ethical ones.  And to really know where these services add value and maybe where they don't, and the role that people need to play in making them work for the organization. So we're going to get into all that today. And with that, let's go over to the presentation and feel free to add questions during the presentation we will be answering those at the end and we look forward to the discussion.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=407.0,456.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/11","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen Before we continue, let's define human centered design so that we can apply it to AI tool evaluation for DAM. Human-centered design is a problem-solving process grounded in empathy and iteration. It's primary focus on empathy is used to uncover needs and insights of users to build an accurate mindset when considering possible solutions. And in order to do this, we need to start with people. The first stage is empathize, in which we want to engage directly with those who will utilize or be directly impacted by a solution so that we can understand their motivations, behaviors and pain points. The second stage is to use what we learn from those people to define. Our goal is to solve the right problem. So here we are, defining a clear problem statement and expected impact. This will guide and focus our work for the remainder of the effort. Our next stage is to ideate or generate lots of potential solutions to the problem. At the outset, no idea is bad, but at some point we want to converge on potential solutions to try out. We do this in the prototype stage. Next, we need to test these prototypes with real users to get feedback and learn if the solution does indeed solve that original problem they helped us define before we finally implement. This process is designed to be iterative so we can repeat and continuously improve until we find the right solution. Before we dive into our AI evaluation framework, I'm going to turn it over to Jason, who's going to set us off with an explanation of what AI for DAM is. Jason.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=456.0,553.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/12","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh Thanks, Kara. In this context, when we say we mean it in a narrow way, we're usually talking about small applications or services that use models of certain kinds of data. These models originate as frameworks or algorithms that are trained to interpret visual or auditory data based on sample data. For example, if you want to train a model to recognize a dog in a picture, it would give it lots of images of dogs from a variety of angles and context and tell it \"Here is the dog.\" The model doesn't know what a dog is, but it can learn that these pixels are probably a dog with some degree of certainty. Models have been trained on a variety of data, celebrities, common objects, words, speech patterns, color, etc. Depending on the vendor, you may be able to choose one of these preexisting models, or you could possibly train a type of model on your own content. For example, say your company does a lot of events and that these events are photographed for resharing or recruiting purposes. It would be helpful to quickly tag the images with the names of significant people that appear in them. You could train your image recognition model to recognize particular individuals that supply those tags. Training a model is not always a fast process, but if you have the need and the training data available, it may be worth the investment. Many organizations using are to trying to get the most out of existing assets by repurposing or monetizing licensing them. Using an AI tool can add fresh new metadata to existing collections, making assets easier to find and reuse for other purposes. AI tools can also enable a hybrid workflow where a tool supplies initial information, which is then validated and enhanced by skilled staff, AI is greater performing tedious, time consuming tasks quickly, such as the first pass of transcription or detecting that air and a few materials. So you don't have to manually scrub through entire recordings for processing. Alternatively, AI can help you chip away at your backlog of digital assets that maybe don't require the same level of attention by applying basic descriptive metadata tags, which would then be validated. Either way, it reduces the time and energy required to make these assets available and accessible in your DAMS. \n\nIf your assets have the potential to contain PII PIFI like names, addresses, credit card numbers or whatever, and AI could quickly identify those and allow you to reduce your exposure by removing, modifying or otherwise restricting those assets, other AI models can help detect fonts, logos, celebrities or other copyrighted license content within files and help you stay within legal compliance. AI models cannot reliably infer contextual information, such as usage rights, project information, photographer, etc., because those metadata aren't necessarily a part of the image and almost always has to come from a related record system or person. An AI model doesn't understand cultural norms or linguistic connotations, pre-trained models have their own biases, and they suggest metadata that is inaccurate, inappropriate or offensive. Like tagging two people together in an image as a couple or a tiger as a cat. Trainable models require you to select the correct data for training, as well as provide feedback for the model. Even pre-trained models sometimes need to be controlled with allow/deny list to prevent specific terms from appearing. And AI tools require human beings to be actively engaged with them to be successful. As such, when either exploring or rigorously evaluating it for the users of the system, whether they be digital asset managers or end users need to be in focus. The AI tools can help speed up workflows without humans in the loop, errors are faster too. AI can label objects that it detects and images and provide descriptive keywords can also identify colors, detect edges of objects for auto cropping and infer mood or sentiment like \"I want nothing but happy dogs.\" AI can also scan for specific entities like celebrities again or branding, and can detect text in an image. For example, a sign at an event.  AI's can automatically translate text into other languages, analyze sentiment to detect positive or negative tone, detect PII or PFI, and even classify or categorize the contents of documents. AI can generate a transcript and identify speakers from an audio track, which could then be used to create captions. AI's can also detect scene or track changes for easier editing and provide information as timestamps. And just like with images and the AI can identify and process objects such as signs using OCR to provide you with the text. \n\nYou may already have identified gaps in your metadata, and automation can maybe help you fill in some of those gaps. Or maybe you're simply curious about, I want to see what's possible. Either way, we recommend that you try one or more services out for yourself. There's a lot of vendors out there and many different out of the box models as well. A good place to start is to ask your DAMS vendor as they likely have one or more AI services available that you can try. There's also free or low cost demos available from AI service providers if you aren't using a DAMS or your provider doesn't natively provide a service. If you've decided AI tools are for you and have selected potential providers and model types, it's important to rigorously evaluate them. After all, you're trusting your metadata to an automation. So you want to ensure that it's meeting your requirements for accuracy while addressing your organization's needs. There's many ways to approach this. We will demonstrate one method that will enable you to make this decision with confidence. \n\nFor the remainder of this presentation, we're going to walk you through a process for evaluating aid solutions, using human centered design as a framework. We'll see how we can work through each of the human centered design stages as we evaluate AI tools for our DAM. In the empathize stage, we identified user needs and motivations as well as technical constraints. In the define stage, we document testable problem statements and define what success should look like.  In the ideate stage we brainstorm possible solutions, benefits and risks of AI for our users. In the prototype stage, we develop ground truth data and run these through candidate solutions. In the test stage, we measured the results of testing against a defined set of evaluation criteria. And finally, once we are ready to implement, we continue to apply these approaches by monitoring, testing and improving. \n\nWe're going to use a hypothetical company, Feed Co., as we navigate each stage of this process. This is a large pet food company that has a DAM with thousands and thousands of image assets. Photos of dogs, cats, even horses and rabbits that their marketing department can use for their advertising. Most of these assets don't have much metadata, so marketing staff have a hard time finding the right images they need for their ad campaigns. If there is a way to tag some of the content in the images, it would save a lot of time spent searching for relevant assets and enable marketing staff to design more effective ads. Feed Co. doesn't have the staff to do this by hand, but they've heard that this might be something AI could help with. Now I'm going to hand it over to Shawn to start walking us through the stages. Shawn.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=553.0,979.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/13","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Thanks, Jason. So in our empathize stage, sometimes called our discovery stage, we want to identify our users and understand their goals and motivations. So we'll get as much information as possible to try to design an AI solution to meet their needs. Be specific when you're talking about the users you're designing for. If the solution is for staff users, what are their roles or titles? If your users are researchers, what kind of researchers are they. In our pet food company example, we're trying to find a solution for users in the marketing department. This target group of users might include marketing design managers, graphic designers or social media managers, and they all will likely have similar goals, but their own specific needs for finding and using assets. To learn about these needs, there's many different tools that we could use from our discovery toolbox, like interviews, workshops or focus groups, which we don't have time to get into today. But the important thing is that we spend some time getting to know these users, their motivations, how they work and how they interact with the DAMS so that we can put ourselves in their shoes as we consider AI potential solutions. \n\nWe also want to be able to understand what types of risk or harm I might pose to our users so that we can be thinking about potential mitigation strategies from the start. Once we've identified the user types that we want to focus on, we've done our discovery activities and we've learned more about our users. We can create some structured user stories and scenarios. Since we often have our own biases and strong ideas for an AI solution that we want to use or for how it might work, this stage always feels like the easiest to gloss over. But I want to emphasize that it's important to spend some quality time here getting a clear picture of the users that will be most affected by the solution. These user stories and scenarios will guide the whole process, and if you do a good job of articulating the needs, goals, motivations and ideal interactions, the rest of the process will be much more straightforward. For user stories, we will use this template that would be familiar to any of you who work with agile software development teams- \"As a type of user....I want to perform some task.... So that I can achieve some goal.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=979.0,1125.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/14","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh This helps us understand the goals of our users and the tasks they need to do to meet those goals. For a Feed Co. marketing department example, we might come up with something like \"As a social media manager, I want to find photos of cats in costumes so I can create some funny tweets for Halloween.  Or as a creative director, I want to find photos of dogs outside so I can find inspiration for our ad campaign promoting our new product line for active dogs. Write as many of these as you can for each of the identified user types. Next, you can create some specific user scenarios for what a successful user story would look like in practice. These can be hypothetical or based on actual scenarios you learned about from your users and your discovery exercises. Imagine these scenarios are happening in a perfect world, even if it's not yet possible in your DAM. \n\nFor our creative director user story, we might come up with the following scenario \"A creative director is working on an ad campaign to promote Feed Co's new product line of food for active dogs. She wants to find as many photos as possible that show dogs outside for inspiration in conceptualizing the campaign, but also for possible use in advertising content. She goes to the Feed Co. DAM and does a keyword search for \"dog\" with tens of thousands of results. She narrows her search further to find dogs outside by adding facets for things you might find outside, like \"trees\" or \"bushes\" or \"fences\" or \"bicycles.\" Satisfied with the abundance of results, she scrolls through the images and saves the ones she likes to a light box. \n\nNote that instead of searching a facet for tags about objects, we could have had our creative director search on \"outside\" to narrow her results to dogs outside. There will often not be just one way for a user to reach their goal, so consider alternative scenarios. None of these are right or wrong. They just represent different requirements for AI solutions in tagging the assets our users want to find. For example, you might need two different AI solutions to support the second scenario- the one that identifies objects in an image like trees or dogs and one that classifies an image as being taken inside or outside. Don't worry too much about the feasibility of implementation yet, though, at this stage, write as many of these scenarios as you can without ruling any of them out in the next stage, we'll look at the use cases for all of the user types together, to find common needs and search strategies that will help us prioritize AI solutions that will serve as many of our users as possible.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=1125.0,1269.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/15","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp In this stage, we also want to talk to our technical team or staff to learn more about some of the local constraints or technical limitations to implementation of an AI technology. This is not an exhaustive list, but these evaluation criteria should cover most of the considerations for assessing the fit of an AI solution for your DAMS. Work with the relevant people within your organization to review each criteria and describe any limitations within each category. Then rank the importance of each one within the list. This will help us in the selection of candidate AI solutions to assess in more depth by allowing us to weed out vendors or technologies that we know won't be feasible for our organization. For example, cost may not be a major limiting factor, but maybe in your department you don't have access to IT support who can help you integrate a commercial cloud service into your workflows, in which case this list might guide you towards a DAM vendor solution. \n\nIn the define stage, we gather all of the information we created in the empathize or discovery stage. Then we analyze it to see if our initial problem statement has changed or can be refined. Look at the goals and motivations that you detailed in your user stories, paying special attention to any commonalities you see in the tasks and see if you can write a problem statement that succinctly describes what you want to solve for your users. You're not going to solve all of their problems right now. But as with implementing any technology, you want the investment to be worthwhile. So make sure that you're putting the purpose first, rather than implementing AI for AI's sake.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=1269.0,1373.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/16","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh In our example, we've only shown two user stories, but let's imagine we've done a full round of discovery and found that most of our marketing users want to be able to find images by the type of animal depicted. Some of them would like to be able to distinguish between breeds of dog or horse or whatever. But for the most part, everyone at least wants to be able to find all of the pictures with a certain type of animal, and they would like to search for other objects and the images to help them narrow their search further. So we could write a problem statement like \"Marketing staff need to be able to search the DAM to find images containing certain types of animals i.e. dog, cat, hamster, ferret and other common objects to efficiently create successful marketing campaigns.\"\n\nIn this stage, we can also start to define measures of success for any solutions to this problem, for each user scenario covered by the problem statement. What would success look like? What quantitative or qualitative methods can you use to measure these successes? We'll use these measures later in the process as we're testing each AI candidate solution. In our creative director example, we could say that success would be finding at least 75 percent of all outside dog photos that exist in the DAM. We also know that our creative director is very busy and does not have time to weed through photos of cats or horses or dogs inside. So let's also say that at least 90 percent of the photos need to be accurate- photos of dogs outside. These measures are totally flexible, obviously, but we want to start with some baseline for quality that we can measure that represents what is important to our users as they're trying to do their jobs.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=1373.0,1466.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/17","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Now that we've defined our problem and set a bar for success, we get to the fun part of ideating or brainstorming ideas and crafting possible solutions. We may have started our process with a specific AI solution in mind, and that might still seem like the most obvious one. But our discovery and definition process may have reframed the problem we thought we wanted to solve, or it may have surfaced new problems that might benefit from a different solution, which may or may not be AI. By now, you might already be familiar with the different kinds of AI and what they do, but if not, now is the time to learn about what's possible and get at least a surface understanding of how the technologies work, so you can be sure that you're selecting the right type of AI for the problem. \n\nIn our example, we've chosen a pretty straightforward problem of detecting objects within an image.  An object detection algorithm is usually a combination of multiple AI algorithms. First, object recognition or localization, finds objects within an image and draws boxes around them. Then object classification looks at each of these boxed objects and classifies them based on all of the kinds of objects that it knows or the model that it was trained on.  For our Feed Co. example, we want to test object detection tools that know how to find dogs, cats versus rabbits, et cetera. \n\nNow that we know what kind of AI solution we want to test for our problem, we should also consider the benefits of such a solution for our users, as well as the risks or potential harms. At AVP, we like to do this in a workshop format involving people knowledgeable of the content or collections, staff that are involved in the creation or management of metadata or assets. Users themselves, especially if they're also staff or staff who interface with the end users of the assets. The more diverse the group involved in this exercise, the better. We want to get as many perspectives as possible to help us understand the impact of candidate AI solutions and to brainstorm ways to mitigate the risk that comes with AI and implement it responsibly. Whether this is in a group setting or just a written exercise done asynchronously, consider the following questions about the actual benefits of this solution. \n\nWhat are the possible benefits to end users and staff supporting or managing the assets? Why does it make more sense to have an AI do this work instead of humans or as part of a human workflow? What do you gain? It's important to confirm the actual benefits of AI for our defined problem, so we make sure that we're not just implementing AI because it's new and shiny. We also need to understand that there will always be some level of risk or potential harm involved when implementing AI tools. Anticipating risks helps to shape an evaluation strategy by identifying the kinds of inaccuracies to test for. Brainstorming possible mitigation strategies can help you envision possible workflows or quality control processes for reducing risks of harm to users. Consider the following questions in evaluating the possible risks. \n\nWhat are the risks to users of your system from implementing AI? What are the risks to the staff who are interacting with the AI output in their workflows? And what are the risks to the subjects or creators of your collections or assets? How might they be misrepresented or misinterpreted? Finally, consider the risks for each user type and their particular user scenario. What could go wrong? How will that impact the user? Which risks are more important to consider than others? In our Feed Co. example, some risks to our creative director might be that they feel their time is wasted, so they end up using photos from an old ad campaign or they lose trust in the management of the DAMS. Once you have a list of risks, brainstorm possible ways to mitigate them. You want to revisit this list throughout the process as you get a sense of the accuracy of and flaws of each AI tool and imagine how you can design workflows that will still benefit from them. \n\nRevisit your evaluation criteria now as you compile a list of possible vendors or AI technologies to test. Who meets your current criteria so far? Accuracy testing of AI solutions can be time consuming. So if you're able to narrow down your candidate list beforehand, you can spend more time testing a wider range of content with a few good tools. If it's possible, you might try out any online demos to make sure that they're producing the right type of results at the level of granularity that you need. For example, if we were hoping to find an out of the box detection tool that identifies dogs by breed, we might be very disappointed to test five candidates and then find out that all of them only detect dogs as dogs. \n\nNow that we have a list of candidate AI solutions to test, we will select some content from our DAM descend through each one and generate output to test.  Select several representative items from your collection or DAM. You will be creating ground truth data or hypothesis data to compare against the AI generated output for testing. So select as many samples as you have time to create ground truth. Let your user stories, success measures and anticipated risks determine what types of content and how many you select. For example, if you're concerned about a speech recognition tool misrepresenting certain dialects of speech in your collections, make sure to select samples representing a wide variety of different dialects so that you can see how well the AI will perform with each. You won't use the ground truth data until the test phase, but it's a good idea to create this before you run your content through the AI tools, at least for more subjective AI tasks like object detection or sentiment analysis, so that you aren't influenced by the results. Going through the process of creating the ground truth data can also help you ensure that you're selecting a diverse range of content for testing. \n\nThe level of ground truth data you need to create will vary by the type of AI tool. Look to your user stories and your measures of success to guide you in creating the appropriate types of ground truth. For example, if you're evaluating video OCR to pull names from the closing credits of the movie to include in your metadata, your ground truth could just be the list of names that you'd expect to find. But if you're evaluating video OCR in order to detect text on signs to include in your closed captioning, you might also want to include the timestamp for each sign text in addition to the text itself, so that you can test to see if the AI tool's detecting every sign.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=1466.0,1899.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/18","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh In our example, we just want to know what objects are in each image. We don't care where they are in the image or how many. So our ground truth for testing object detection could just be a list of words per image. This is a pretty easy ground truth task, as we're just naming things in a photo so we can do a lot of these for testing. Other types of ground truth, like automated speech recognition for audio or scene detection and video will take a lot longer, so you may have fewer assets to test. The hardest part about our task is that we would like to include images of dogs outside, but we have to find some in our DAM without any metadata. When you don't know what you have, it's a good idea to work with someone in your organization who's familiar with the collections or assets so you can get a more representative sample and surfaces many types of AI errors in the testing as possible.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=1899.0,1948.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/19","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Now that we have our sample content selected, we will run them through each candidate AI tool. This will probably be different for each tool. Some vendors offer easy to use interfaces to try out your own content with their service. Others will require some knowledge of scripting or interacting with an API. Try out different settings if those are available. Or try a different resolutions of your sample assets to see how and if it affects the quality of the output. Make sure you document any settings or parameters that you set and save the output in a structured format like CSV or JSON, if possible. You'll use this for your ground truth testing in the test phase. Complete the evaluation criteria for each AI candidate. How much does it cost? How long did it take you to process each of your samples? What extra features or options does each tool offer? What are the input and output requirements? How will this tool integrate into your workflows or existing systems, etc.? Now that you have some results, it's important to measure the quality of them. There are various degrees of quantitative and qualitative testing that you can do. \n\nThe first thing you will need to do is get the data from the output formats of each AI tool into the same format as your ground truth data. This will be easy for our example, since we just have a list of objects that we want to find in a photo. For others, this may need to be a bit more structured, especially if you're working with time based media. Your ground truth might consist of some kind of label at a certain timestamp and CSV files or spreadsheets can be good structures for this kind of ground truth. For most types of AI tasks, you can calculate commonly used metrics based on the numbers of correct and incorrect matches. \n\nWe don't have enough time to get into the details of this process, but basically you'll compare the AI results against your ground truth, noting any true positives which are correct matches any false positives, which are instances where the AI made bad guesses or false negatives, instances where the eye did not detect something from our ground truth. With these numbers, we can calculate metrics like precision and recall. Precision tells us how accurate a high was in matching up with our ground truth. High precision means that the AI did a good job of only matching with the ground truth without generating too many false positives. Recall tells us how well the AI did in finding as many true positives as possible, even if it also identifies a lot of false positives. In our example, we can see that vendor Y's object detection tool matched two of the ground truth labels, but missed three of them and got two of them wrong, resulting in a precision score of point five and a recall of point four. Vendor Z did a little bit better, only producing one false positive for a precision score of point six seven and a recall of point four. \n\nRemember that for our use case, our creative director was concerned about wasting too much time weeding through irrelevant or inaccurate results. So in our assessment, we may want a preference to AI tools that had higher precision since they produced fewer false positives and our ground truth testing. The metrics that matter for you will depend on your organization's particular user needs. So it's important to test this out yourself on your own content rather than just accepting the metrics given by the AI vendor. These quantitative metrics can be helpful in comparing AI tools against each other, but they shouldn't be the only quality assessment you use. Digging deeper into the true positives, false negatives and false positives, to see where the AI did well and where it failed, can help you better assess the levels of risk that you identified earlier and to consider the types of mitigation strategies or human mediation that you might have to apply in an implementation. \n\nShawn Averkamp Again, let your user stories and your measures of success guide you in selecting the best methods to assess the quality of the outputs. For example, in all of our use cases for Feed Co. marketing, we know that it's important for the AI to be able to detect animals looking at Vendor X, which had higher precision and recall. So on the surface might seem like the best choice. We can see that it did a great job of detecting many kinds of objects, just not dogs. Taking the time for this qualitative review saved us from making a big investment in an AI tool that may have given us disappointing results. \n\nAfter you've completed this testing and analysis and assessed how each air candidate stands up against your full evaluation criteria, you may find that you have a clear winner or maybe even more than one to choose from. If so, congratulations. However, it may be just as likely that none of your candidates meet your bar for accuracy, so you may need to iterate and try other candidate AI tools or try more samples or different settings. Or this process might have inspired new ideas for how to use AI to improve the efficiency of an existing workflow, or even ideas for enhancing access to your assets that you didn't imagine at the start. The important thing in testing new and evolving technologies like AI, is to be open to the unexpected uses that may surface as you're testing something else. Once you've chosen an AI tool to implement in your DAM, you should plan on continuing to iterate to improve your data and workflows and to mitigate the risks you identified earlier. \n\nIf you're using a commercial AI vendor, it's important to test regularly, as their algorithms and models are usually opaque to users that are constantly being changed and refined in an effort to improve quality. Hopefully, this will result in higher quality results for your DAM, too. But there may always be unintended consequences, especially if the vendors use cases are different from your own. You can use your ground truth tests as benchmarks for future evaluation by running them in your system on a regular schedule to see if anything has changed. Once you've introduced this new data into your DAM, you might consider doing some user testing to see if and how these AI contributions are helping your users and how you might improve them. User testing can also help you to confirm any assumptions you've made about risk by revealing any harmful or inaccurate data that might need to be further mitigated through new strategies. We know that AI will never be as good as humans at making nuanced judgments and creating appropriate metadata, but by always keeping our users and risks in focus, we can responsibly use AI tools to our advantage.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=1948.0,2374.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/20","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen Thank you so much, Shawn and Jason. That was awesome. Before we go to Q\u0026A, I just want to remind you that at AVP we're here to help. We could do a workshop with you and plan areas where AI can add value to your DAM program. We can support you in a full AI evaluation process, as we've outlined here. Or we could assist with the AI implementation, including mapping human and machine workflows, modeling data, structuring AI outputs for utilization, integrations, all sorts of things. \n\nI also want to invite you to apply for a free online workshop that we're doing for up to 20 people. We're going to be holding this in April, and the goal is to teach you how to conduct user research within your organization to optimize your DAM experience. You can find a link to sign up for that on our blog at weareavp.com and you can see the URL to sign up right here on the screen. All right, it's time for Q\u0026A, so I'm going to hand it over to Jeanette. Thank you very much.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2374.0,2434.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/21","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thank you very much. If you'd like to come back in for Q\u0026A, we have had some questions. Fabulous, thank you. Someone has asked, \"Have you found much difference in quality in open source tools and vendor solutions?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2434.0,2455.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/22","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp I can take that one. We did test a number of different commercial tools and open-source tools with the AMMPD project. And I have to say it really depends on the type of tool. I think an important thing to look at when you're evaluating these tools is what is the use case or the model that the vendor is using for their AI tool. So, for example, a vendor might be training their model on call center data or they might be training it for the purpose of search. So, you know, really looking at what that model was ideally used for can kind of give you a window into how well that might work with your content. It really depends on your contents. Certainly, some tools just start a little bit better than others and have been trained on a lot more data. But yes, looking at the use cases for the vendors or the open-source tools. And if you're able to, and this is the perk of open source tools, is that you usually can see the model of the type of content that the tool was trained on, as well as maybe from a different community forums or documentation. You can see why the tool was created in the first place.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2455.0,2542.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/23","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thank you. And someone else's asked \"How common are offensive or inappropriate terms, are the models improving in this area? And what kinds of inappropriate results might I come across?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2542.0,2562.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/24","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Do you want me to take that one or Jason, have you come across anything particularly egregious?","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2562.0,2567.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/25","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh before it's it's tricky question, because inappropriate is super local to you sometimes like I was saying in the example to people on a picture is not necessarily a couple. So you just kind of want to keep an eye on things like that, like offensive content is mostly dependent on what the model's been trained on, like you were just saying, like what kind of data they have been feeding it. And if it gives you objectionable results, you should either examine like a different model or or have some way of filtering those things out. But I think by and large, like as more data as these models are trained on more sample assets, they're getting better, especially the ones from the large vendors. They're like pretty good at not doing anything that most people would not want.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2567.0,2610.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/26","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp I think something that I've seen that I think is especially important for those of you who are working with older assets. So any of you who manage archives or libraries is a lot of tools, especially vendor tools have been trained sometimes on popular culture, people and terms. And so you might find it's calling certain people current celebrities or maybe speech recognition. Is this is one that we saw was it kept finding Harry Potter in a lecture from the nineteen sixties, which obviously was not Harry Potter. So I think timeliness, you know, is something to be aware of. And while it may not be offensive, it is certainly inaccurate and it may, you know, give your users pause about how much they trust your archive, depending on how you're presenting that data to the users. So I think. Yeah, certainly keeping an eye out for things that are inaccurate and inappropriate, in addition to offensive, it's something to be wary.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2610.0,2689.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/27","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping OK. Someone's asked, \"What kind of scale do you typically use for testing? Dozens, hundreds, thousands, tens of thousands of images? \"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2689.0,2702.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/28","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp It really depends on what you have time to create ground truth for. We've recently worked with a client who worked with text documents. And for that, we were able to create ground truth for about 30 or so documents, which I think gave us a pretty good, good sense of how the tools worked. For the AMP project, working with time based media is certainly much more labor intensive to create ground truth, especially if you need to record time codes. I'd love to see more tools out there that will help users do this, but we're now finding in this third phase as we're starting to package and optimize the system. We're working with new collections and we're finding a lot of new errors. Just working with new collections and more media, which is great to surface as that. Yes, keep in mind that especially with time based collections, there will be tradeoffs in time and then accuracy.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2702.0,2766.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/29","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh And to follow that up, depending on your vendor, if you're doing a demo, there may be a limit on how many images or whatever you run through the system, how much data. So like kind of carefully curating what you're submitting to being like a representative sample, particularly if there's like anything you have a lot of like similar content like event photography or building photography or whatever that you make sure you test those things so that you get like results that are meaningful to you to.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2766.0,2795.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/30","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping But thank you. Next question, \"How should one go about identifying AI vendors to evaluate? Is there a resource that lists many or most vendors in one place? Googling seems hit or miss.\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2795.0,2814.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/31","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Jason, I know you've done a bit of searching around on this topic.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2814.0,2820.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/32","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh I knew who some of the vendors were from past experience, but I did not come across anything that was like a clearinghouse for different services or vendors. It's it also really depends on your system configuration. Sorry about that. So if you're if you're already in a DAM system, they may or may not support particular services or integrations. And if you're rolling your own, you maybe have more options. But there are some. There are some, like large vendors out there that will immediately come up when you search for them, which I'm not going to mention here. But they are out there and they're pretty easy to find if you want to at least get a sense of what you're looking at.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2820.0,2865.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/33","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping OK, just seeing if there's any more questions from anybody, now's the time to ask them what they're coming to the end of them. I just want to remind everybody that we're sending a link to the recording, probably tomorrow, so you'll all get back to you again at your own pace. Right. We have had another question, thank you. \"How do you handle descriptors of gender or nationality race, given that the current trend is to be more open ended rather than concrete tax (onomy)?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2865.0,2902.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/34","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen I have a little bit of a thought on this that comes from not the AI evaluation experience, but from user testing with DAMS just generally. So when we work with users to see how they search and behave, when they're actually interacting with DAM and looking for assets, they are searching for terms like woman and man, like they're they're looking for something that will help them create whatever it is they're creating. And they in their private searching moments will use terms that will help them get to those results. So this is just something that we have found in our experience that this is just how people behave and when they. But how do you then create metadata that kind of, you know, is going to not gender or, you know, identify people necessarily in a way that you don't feel is appropriate. So a lot of systems will allow you to have synonyms or terms that are hidden that you're not necessarily displaying to the users and that you can have your taxonomy. That's, you know, as you'd like it to be, as you'd like those people to be represented as your front facing. But I think that's one thing to keep in mind is that the way that you want to describe and represent isn't always the same as how someone is really searching. And I know that's a delicate balance, but it is something to keep in mind, and you can take the benefit of those more hidden terms to really support that search and drive the search to help people find what they're looking for, but then show them how to represent that appropriately once they've found the assets. So I'll just share that thought, but I'll see if Shawn or Jason have any thoughts based on their experience with the output of the AI tools.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=2902.0,3006.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/35","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Yeah, I think it's important in this case, as always, not just with AI but to make sure you understand if you need and why you need certain metadata about gender or race or nationality. One of the tools that we worked with on the AMP project does segmentation of audio, so it will segment out speech, silence, music, noise. And it also will identify for speech segments. It will try to identify the gender. So it just automatically gives you that data. Whether or not you use it, that's up to you. So the original intent of this tool was actually to build an app, to look at conversations and to see if they were male dominated or female dominated. Women dominated. And so for that original purpose of AI, it was very important for them to use that but for our purposes, we did not see the need to include gender in the tool. So just because an AI tool gives you certain information, does not that mean that you have to use it.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3006.0,3076.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/36","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh And to follow up on both of those threads, if you are running an AI tool you might you probably want to have those tags go to a field that is not displayed to people, at least while you're figuring things out just in case it does something wrong so that you have a chance to filter and feed on that. And then, too, you can forbid certain words. If you don't think it's important to tag a woman or man in a picture, just just, you know, deny it. Just tell it. Don't do that. If you have that capability, it will save you some trouble later. But.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3076.0,3110.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/37","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thank you. And I think this is the last question, and it \"Is the conversion of AI tags to real tags, a normal part of workflow?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3110.0,3125.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/38","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh This is kind of what I was just talking about, so in in most cases, I think that you may not want to allow an automation to put things on content that are going to immediately be displayed to users, at least until you've had a chance to, like, validate it check it out. Whether or not it gets promoted to a real tag is sort of up to you. I don't think there's any risk in leaving it where it goes, because usually the automation will want to map to a particular field, depending on how it's set up. At least that's true for like keywords. I don't know about other A.I. types, like if there's a need to like, promote it, or sometimes that's the only tag you're getting. So in which case you may want to think about having some kind of workflow to promote it, but.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3125.0,3173.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/39","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Amazing, thank you. Well, that seems to be the end of the questions, I think if there were a couple more would follow. We can follow up offline with you. We haven't had a chance to get to all of them. I just would like to thank all of you very much indeed. Have you got anything else to say before we leave for the day?","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3173.0,3192.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/40","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen We just thank you so much for joining us. I think we're going to share this recording and a transcript as well, so be on the lookout for that and feel free to follow up with any questions you might have that we didn't get to shoot us an email. We're here to help. So happy to answer any questions. Thank you so much.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3192.0,3214.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647/transcript/35948/annotation/41","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thanks, everybody. We'll see you on another webinar soon. By now.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154647#t=3214.0,3222.50667"}]}]},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667","type":"Canvas","label":{"en":["Media File 2 of 2 - 1648151119Finding_the_Right_AI_Tools_for_DAM_AVPs_Human_Centered_Evaluation_Framework20240423-104915-9eo894.mp4"]},"duration":800.027,"width":640,"height":360,"thumbnail":[{"id":"https://d9jk7wjtjpu5g.cloudfront.net/collection_resource_files/thumbnails/000/154/667/small/1648151119Finding_the_Right_AI_Tools_for_DAM_AVPs_Human_Centered_Evaluation_Framework.mp4_1648152134.jpg?1648137748","type":"Image","format":"image/jpeg"}],"items":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/content/1","type":"AnnotationPage","items":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/content/2/annotation/1","type":"Annotation","motivation":"painting","body":{"id":"https://aviary-p-weareavp.s3.wasabisys.com/collection_resource_files/resource_files/000/154/667/original/1648151119Finding_the_Right_AI_Tools_for_DAM_AVPs_Human_Centered_Evaluation_Framework20240423-104915-9eo894.mp4?1713862963","type":"Video","format":"video/mp4","duration":800.027,"width":640,"height":360},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667","metadata":[]}]}],"annotations":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949","type":"AnnotationPage","label":{"en":["Transcript AVP Finding the Right AI Tools for DAM [Transcript]"]},"items":[{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/1","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen Thank you so much, Shawn and Jason. That was awesome. Before we go to Q\u0026A, I just want to remind you that at AVP we're here to help. We could do a workshop with you and plan areas where AI can add value to your DAM program. We can support you in a full AI evaluation process, as we've outlined here. Or we could assist with the AI implementation, including mapping human and machine workflows, modeling data, structuring AI outputs for utilization, integrations, all sorts of things.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=0.0,2.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/2","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thank you very much. If you'd like to come back in for Q\u0026A, we have had some questions. Fabulous, thank you. Someone has asked, \"Have you found much difference in quality in open source tools and vendor solutions?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=2.0,23.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/3","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp I can take that one. We did test a number of different commercial tools and open-source tools with the AMMPD project. And I have to say it really depends on the type of tool. I think an important thing to look at when you're evaluating these tools is what is the use case or the model that the vendor is using for their AI tool. So, for example, a vendor might be training their model on call center data or they might be training it for the purpose of search. So, you know, really looking at what that model was ideally used for can kind of give you a window into how well that might work with your content. It really depends on your contents. Certainly, some tools just start a little bit better than others and have been trained on a lot more data. But yes, looking at the use cases for the vendors or the open-source tools. And if you're able to, and this is the perk of open source tools, is that you usually can see the model of the type of content that the tool was trained on, as well as maybe from a different community forums or documentation. You can see why the tool was created in the first place.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=23.0,110.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/4","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thank you. And someone else's asked \"How common are offensive or inappropriate terms, are the models improving in this area? And what kinds of inappropriate results might I come across?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=110.0,130.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/5","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Do you want me to take that one or Jason, have you come across anything particularly egregious?","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=130.0,135.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/6","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh before it's it's tricky question, because inappropriate is super local to you sometimes like I was saying in the example to people on a picture is not necessarily a couple. So you just kind of want to keep an eye on things like that, like offensive content is mostly dependent on what the model's been trained on, like you were just saying, like what kind of data they have been feeding it. And if it gives you objectionable results, you should either examine like a different model or or have some way of filtering those things out. But I think by and large, like as more data as these models are trained on more sample assets, they're getting better, especially the ones from the large vendors. They're like pretty good at not doing anything that most people would not want.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=135.0,178.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/7","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp I think something that I've seen that I think is especially important for those of you who are working with older assets. So any of you who manage archives or libraries is a lot of tools, especially vendor tools have been trained sometimes on popular culture, people and terms. And so you might find it's calling certain people current celebrities or maybe speech recognition. Is this is one that we saw was it kept finding Harry Potter in a lecture from the nineteen sixties, which obviously was not Harry Potter. So I think timeliness, you know, is something to be aware of. And while it may not be offensive, it is certainly inaccurate and it may, you know, give your users pause about how much they trust your archive, depending on how you're presenting that data to the users. So I think. Yeah, certainly keeping an eye out for things that are inaccurate and inappropriate, in addition to offensive, it's something to be wary.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=178.0,257.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/8","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping OK. Someone's asked, \"What kind of scale do you typically use for testing? Dozens, hundreds, thousands, tens of thousands of images? \"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=257.0,270.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/9","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp It really depends on what you have time to create ground truth for. We've recently worked with a client who worked with text documents. And for that, we were able to create ground truth for about 30 or so documents, which I think gave us a pretty good, good sense of how the tools worked. For the AMP project, working with time based media is certainly much more labor intensive to create ground truth, especially if you need to record time codes. I'd love to see more tools out there that will help users do this, but we're now finding in this third phase as we're starting to package and optimize the system. We're working with new collections and we're finding a lot of new errors. Just working with new collections and more media, which is great to surface as that. Yes, keep in mind that especially with time based collections, there will be tradeoffs in time and then accuracy.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=270.0,334.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/10","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh And to follow that up, depending on your vendor, if you're doing a demo, there may be a limit on how many images or whatever you run through the system, how much data. So like kind of carefully curating what you're submitting to being like a representative sample, particularly if there's like anything you have a lot of like similar content like event photography or building photography or whatever that you make sure you test those things so that you get like results that are meaningful to you to.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=334.0,363.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/11","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping But thank you. Next question, \"How should one go about identifying AI vendors to evaluate? Is there a resource that lists many or most vendors in one place? Googling seems hit or miss.\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=363.0,382.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/12","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Jason, I know you've done a bit of searching around on this topic.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=382.0,388.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/13","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh I knew who some of the vendors were from past experience, but I did not come across anything that was like a clearinghouse for different services or vendors. It's it also really depends on your system configuration. Sorry about that. So if you're if you're already in a DAM system, they may or may not support particular services or integrations. And if you're rolling your own, you maybe have more options. But there are some. There are some, like large vendors out there that will immediately come up when you search for them, which I'm not going to mention here. But they are out there and they're pretty easy to find if you want to at least get a sense of what you're looking at.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=388.0,433.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/14","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping OK, just seeing if there's any more questions from anybody, now's the time to ask them what they're coming to the end of them. I just want to remind everybody that we're sending a link to the recording, probably tomorrow, so you'll all get back to you again at your own pace. Right. We have had another question, thank you. \"How do you handle descriptors of gender or nationality race, given that the current trend is to be more open ended rather than concrete tax (onomy)?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=433.0,470.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/15","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen I have a little bit of a thought on this that comes from not the AI evaluation experience, but from user testing with DAMS just generally. So when we work with users to see how they search and behave, when they're actually interacting with DAM and looking for assets, they are searching for terms like woman and man, like they're they're looking for something that will help them create whatever it is they're creating. And they in their private searching moments will use terms that will help them get to those results. So this is just something that we have found in our experience that this is just how people behave and when they. But how do you then create metadata that kind of, you know, is going to not gender or, you know, identify people necessarily in a way that you don't feel is appropriate. So a lot of systems will allow you to have synonyms or terms that are hidden that you're not necessarily displaying to the users and that you can have your taxonomy. That's, you know, as you'd like it to be, as you'd like those people to be represented as your front facing. But I think that's one thing to keep in mind is that the way that you want to describe and represent isn't always the same as how someone is really searching. And I know that's a delicate balance, but it is something to keep in mind, and you can take the benefit of those more hidden terms to really support that search and drive the search to help people find what they're looking for, but then show them how to represent that appropriately once they've found the assets. So I'll just share that thought, but I'll see if Shawn or Jason have any thoughts based on their experience with the output of the AI tools.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=470.0,574.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/16","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Shawn Averkamp Yeah, I think it's important in this case, as always, not just with AI but to make sure you understand if you need and why you need certain metadata about gender or race or nationality. One of the tools that we worked with on the AMP project does segmentation of audio, so it will segment out speech, silence, music, noise. And it also will identify for speech segments. It will try to identify the gender. So it just automatically gives you that data. Whether or not you use it, that's up to you. So the original intent of this tool was actually to build an app, to look at conversations and to see if they were male dominated or female dominated. Women dominated. And so for that original purpose of AI, it was very important for them to use that but for our purposes, we did not see the need to include gender in the tool. So just because an AI tool gives you certain information, does not that mean that you have to use it.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=574.0,644.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/17","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh And to follow up on both of those threads, if you are running an AI tool you might you probably want to have those tags go to a field that is not displayed to people, at least while you're figuring things out just in case it does something wrong so that you have a chance to filter and feed on that. And then, too, you can forbid certain words. If you don't think it's important to tag a woman or man in a picture, just just, you know, deny it. Just tell it. Don't do that. If you have that capability, it will save you some trouble later. But.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=644.0,678.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/18","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thank you. And I think this is the last question, and it \"Is the conversion of AI tags to real tags, a normal part of workflow?\"","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=678.0,693.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/19","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jason Ulsh This is kind of what I was just talking about, so in in most cases, I think that you may not want to allow an automation to put things on content that are going to immediately be displayed to users, at least until you've had a chance to, like, validate it check it out. Whether or not it gets promoted to a real tag is sort of up to you. I don't think there's any risk in leaving it where it goes, because usually the automation will want to map to a particular field, depending on how it's set up. At least that's true for like keywords. I don't know about other A.I. types, like if there's a need to like, promote it, or sometimes that's the only tag you're getting. So in which case you may want to think about having some kind of workflow to promote it, but.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=693.0,741.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/20","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Amazing, thank you. Well, that seems to be the end of the questions, I think if there were a couple more would follow. We can follow up offline with you. We haven't had a chance to get to all of them. I just would like to thank all of you very much indeed. Have you got anything else to say before we leave for the day?","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=741.0,760.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/21","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Kara Van Malssen We just thank you so much for joining us. I think we're going to share this recording and a transcript as well, so be on the lookout for that and feel free to follow up with any questions you might have that we didn't get to shoot us an email. We're here to help. So happy to answer any questions. Thank you so much.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=760.0,782.0"},{"id":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667/transcript/35949/annotation/22","type":"Annotation","motivation":"transcribing","body":{"type":"TextualBody","value":"Jeanette Camping Thanks, everybody. We'll see you on another webinar soon. By now.","format":"text/plain"},"target":"https://weareavp.aviaryplatform.com/collections/6/collection_resources/70011/file/154667#t=782.0,790.506"}]}]}]}