[{"data":1,"prerenderedAt":1160},["ShallowReactive",2],{"\u002Fpost\u002Fbasic-content-filter":3},{"id":4,"title":5,"body":6,"createdAt":1152,"description":1153,"extension":1154,"meta":1155,"navigation":461,"path":1156,"seo":1157,"stem":1158,"__hash__":1159},"posts\u002Fpost\u002Fbasic-content-filter.md","I added a basic content filter and learned something nifty",{"type":7,"value":8,"toc":1132},"minimark",[9,14,18,21,24,28,31,44,48,59,64,67,70,74,77,80,84,87,91,94,105,108,114,125,131,135,138,268,275,281,285,288,335,339,342,902,906,909,915,919,922,925,929,932,1022,1025,1034,1040,1044,1047,1050,1053,1062,1065,1068,1075,1078,1083,1094,1100,1104,1107,1113,1122,1125,1128],[10,11,13],"h2",{"id":12},"introduction","Introduction",[15,16,17],"p",{},"Recently I thought to myself, \"Since my side project is linked on my resume and LinkedIn profile, it would probably be bad if somebody uploaded porn to it.\"",[15,19,20],{},"Content filtering is not a serious concern to me at this time because my app has no users other than myself. But with that said, it's still technically possible for a random stranger to log into topical.space, make an account and upload pictures of... whatever. So I felt a little bit nervous about tying my professional reputation to a website that allows user-generated content without a filter.",[15,22,23],{},"I decided to write this post because I found a solution that I think is pretty cool, since it filters content on the Google Cloud Platform side without actually requiring any code changes to my app.",[10,25,27],{"id":26},"current-image-upload-process","Current Image Upload Process",[15,29,30],{},"For context, here's how image uploads currently work in topical.space:",[32,33,34,38,41],"ol",{},[35,36,37],"li",{},"A user clicks Add Image or pastes an image into the text editor.",[35,39,40],{},"The client calls a backend resolver called createSignedStorageURL. The backend uses\nits Google Cloud credentials to call the Google Cloud Storage API to create a special URL.\nThis special URL, called the \"signed storage URL\", is the URL where the image\nwill be available after it is uploaded to GCS by the client. It comes with permissions\nfor the client to upload an image to that URL for a limited amount of time.",[35,42,43],{},"The client calls the GCS API to upload the image to the signed storage URL.",[10,45,47],{"id":46},"implementing-the-content-filter","Implementing the Content Filter",[15,49,50,51,58],{},"So I started researching automated image scanners, with the intent of trying to make it difficult for any potentially explicit user-generated content to see the light of day. I figured that since I already hosted images on GCS (Google Cloud Storage), I may as well try another Google service, ",[52,53,57],"a",{"href":54,"rel":55},"https:\u002F\u002Fcloud.google.com\u002Fvision\u002Fdocs\u002Fdetecting-safe-search",[56],"nofollow","SafeSearch,"," for image scanning.",[60,61,63],"h3",{"id":62},"initial-considerations","Initial Considerations",[15,65,66],{},"At first I thought that when the client calls createSignedStorageURL, that should trigger an event that causes my backend app to poll the signed storage URL on GCS to see if an image was uploaded there. If it existed, then the backend would order a scan on the image at that URL. If it didn't exist, the polling would eventually time out.",[15,68,69],{},"But then my question was \"How long should the backend poll GCS until it times out?\" I didn't like the idea of setting the time to an arbitrary limit like 30 seconds or a minute, because then an image could bypass the content filter if the user had a slow connection and it took a long time to upload the picture.",[60,71,73],{"id":72},"the-gcs-event-based-solution","The GCS Event-Based Solution",[15,75,76],{},"So then I asked another question: \"How hard would it be to make GCS itself trigger the scan as soon as an image is uploaded?\"",[15,78,79],{},"And the answer is, it's easy! With a few commands, it's possible to make an event get emitted to Google's event bus system when a file is uploaded to GCS. And it is also easy to use a Google Cloud Function that listens for that event and executes when that event happens.",[10,81,83],{"id":82},"setting-up-the-cloud-function","Setting Up the Cloud Function",[15,85,86],{},"Since I already had the gcloud CLI installed, I set up the content filter with gcloud commands, which is a better user experience than clicking around in the cloud console.",[60,88,90],{"id":89},"creating-the-event-pipeline","Creating the Event Pipeline",[15,92,93],{},"First I created a pub\u002Fsub topic like this:",[95,96,101],"pre",{"className":97,"code":99,"language":100},[98],"language-text","gcloud pubsub topics create image-uploads-topic\n","text",[102,103,99],"code",{"__ignoreMap":104},"",[15,106,107],{},"Then I linked the pub\u002Fsub topic to my GCS bucket:",[95,109,112],{"className":110,"code":111,"language":100},[98],"gcloud storage buckets notifications create gs:\u002F\u002Flistical-dev \\\n  --topic=image-uploads-topic \\\n  --event-types=OBJECT_FINALIZE\n",[102,113,111],{"__ignoreMap":104},[15,115,116,117,120,121,124],{},"Then I made a cloud function called ",[102,118,119],{},"scanImage"," get triggered by ",[102,122,123],{},"image-uploads-topic",":",[95,126,129],{"className":127,"code":128,"language":100},[98],"gcloud functions deploy scanImage \\\n  --runtime nodejs20 \\\n  --trigger-topic=image-uploads-topic \\\n  --allow-unauthenticated\n",[102,130,128],{"__ignoreMap":104},[60,132,134],{"id":133},"implementing-the-cloud-function","Implementing the Cloud Function",[15,136,137],{},"Then I made the cloud function in a local directory. This cloud function is what scans the newly uploaded file and will delete the image if it doesn't pass the content filter. The Google SafeSearch scanner responds in this format:",[95,139,143],{"className":140,"code":141,"language":142,"meta":104,"style":104},"language-json shiki shiki-themes github-light github-dark","{\n  \"responses\": [\n    {\n      \"safeSearchAnnotation\": {\n        \"adult\": \"UNLIKELY\",\n        \"spoof\": \"VERY_UNLIKELY\",\n        \"medical\": \"VERY_UNLIKELY\",\n        \"violence\": \"LIKELY\",\n        \"racy\": \"POSSIBLE\"\n      }\n    }\n  ]\n}\n","json",[102,144,145,154,164,170,179,195,208,220,233,244,250,256,262],{"__ignoreMap":104},[146,147,150],"span",{"class":148,"line":149},"line",1,[146,151,153],{"class":152},"sVt8B","{\n",[146,155,157,161],{"class":148,"line":156},2,[146,158,160],{"class":159},"sj4cs","  \"responses\"",[146,162,163],{"class":152},": [\n",[146,165,167],{"class":148,"line":166},3,[146,168,169],{"class":152},"    {\n",[146,171,173,176],{"class":148,"line":172},4,[146,174,175],{"class":159},"      \"safeSearchAnnotation\"",[146,177,178],{"class":152},": {\n",[146,180,182,185,188,192],{"class":148,"line":181},5,[146,183,184],{"class":159},"        \"adult\"",[146,186,187],{"class":152},": ",[146,189,191],{"class":190},"sZZnC","\"UNLIKELY\"",[146,193,194],{"class":152},",\n",[146,196,198,201,203,206],{"class":148,"line":197},6,[146,199,200],{"class":159},"        \"spoof\"",[146,202,187],{"class":152},[146,204,205],{"class":190},"\"VERY_UNLIKELY\"",[146,207,194],{"class":152},[146,209,211,214,216,218],{"class":148,"line":210},7,[146,212,213],{"class":159},"        \"medical\"",[146,215,187],{"class":152},[146,217,205],{"class":190},[146,219,194],{"class":152},[146,221,223,226,228,231],{"class":148,"line":222},8,[146,224,225],{"class":159},"        \"violence\"",[146,227,187],{"class":152},[146,229,230],{"class":190},"\"LIKELY\"",[146,232,194],{"class":152},[146,234,236,239,241],{"class":148,"line":235},9,[146,237,238],{"class":159},"        \"racy\"",[146,240,187],{"class":152},[146,242,243],{"class":190},"\"POSSIBLE\"\n",[146,245,247],{"class":148,"line":246},10,[146,248,249],{"class":152},"      }\n",[146,251,253],{"class":148,"line":252},11,[146,254,255],{"class":152},"    }\n",[146,257,259],{"class":148,"line":258},12,[146,260,261],{"class":152},"  ]\n",[146,263,265],{"class":148,"line":264},13,[146,266,267],{"class":152},"}\n",[15,269,270,271,274],{},"Therefore, the cloud function checks the ",[102,272,273],{},"safeSearchAnnotation"," in the SafeSearch response to decide whether to delete the file or not. I set up the file structure of my cloud function like this:",[95,276,279],{"className":277,"code":278,"language":100},[98],"scanImage\n  - index.js\n  - package.json\n",[102,280,278],{"__ignoreMap":104},[60,282,284],{"id":283},"package-configuration","Package Configuration",[15,286,287],{},"The package.json adds the Google Cloud Storage and Vision SDKs as dependencies:",[95,289,291],{"className":140,"code":290,"language":142,"meta":104,"style":104},"{\n    \"dependencies\": {\n      \"@google-cloud\u002Fstorage\": \"^6.9.0\",\n      \"@google-cloud\u002Fvision\": \"^3.0.0\"\n    }\n  }\n  \n",[102,292,293,297,304,316,326,330],{"__ignoreMap":104},[146,294,295],{"class":148,"line":149},[146,296,153],{"class":152},[146,298,299,302],{"class":148,"line":156},[146,300,301],{"class":159},"    \"dependencies\"",[146,303,178],{"class":152},[146,305,306,309,311,314],{"class":148,"line":166},[146,307,308],{"class":159},"      \"@google-cloud\u002Fstorage\"",[146,310,187],{"class":152},[146,312,313],{"class":190},"\"^6.9.0\"",[146,315,194],{"class":152},[146,317,318,321,323],{"class":148,"line":172},[146,319,320],{"class":159},"      \"@google-cloud\u002Fvision\"",[146,322,187],{"class":152},[146,324,325],{"class":190},"\"^3.0.0\"\n",[146,327,328],{"class":148,"line":181},[146,329,255],{"class":152},[146,331,332],{"class":148,"line":197},[146,333,334],{"class":152},"  }\n",[60,336,338],{"id":337},"cloud-function-code","Cloud Function Code",[15,340,341],{},"And the index.js of the cloud function is as follows (with a note to remind my future self what I'm looking at):",[95,343,347],{"className":344,"code":345,"language":346,"meta":104,"style":104},"language-js shiki shiki-themes github-light github-dark","\u002F\u002F This file is used for setting up a cloud function via the gcloud CLI.\n\u002F\u002F I used it to set up the function to be triggered by a Pub\u002FSub message,\n\u002F\u002F specifically the message that is triggered by a new file being uploaded to\n\u002F\u002F a GCS bucket.\nconst { Storage } = require('@google-cloud\u002Fstorage');\nconst vision = require('@google-cloud\u002Fvision');\nconst storage = new Storage();\nconst client = new vision.ImageAnnotatorClient();\n\nexports.scanImage = async (event, context) => {\n  const data = JSON.parse(Buffer.from(event.data, 'base64').toString());\n  const bucketName = data.bucket;\n  const fileName = data.name;\n\n  try {\n    \u002F\u002F This is where we have the newly uploaded image scanned by Google SafeSearch.\n    const [result] = await client.safeSearchDetection(`gs:\u002F\u002F${bucketName}\u002F${fileName}`);\n    const safeSearch = result.safeSearchAnnotation;\n\n    if (\n        safeSearch.adult === 'VERY_LIKELY' || \n        safeSearch.adult === 'LIKELY' ||\n        safeSearch.racy === 'VERY_LIKELY' ||\n        safeSearch.medical === 'VERY_LIKELY' ||\n        safeSearch.violence === 'VERY_LIKELY' ||\n        safeSearch.violence === 'LIKELY'\n    ) {\n      \u002F\u002F And this is where we delete an image that doesn't pass the check.\n      await storage.bucket(bucketName).file(fileName).delete();\n\n      \u002F\u002F These logs show in the Google cloud console whenever the cloud function\n      \u002F\u002F is triggered.\n      \u002F\u002F In the future I will also update this area of the code so that it triggers a \n      \u002F\u002F notification, which will tell the user that their image was deleted because it didn't \n      \u002F\u002F pass the content filter.\n      console.log(`Deleted unsafe image: ${fileName}`);\n    } else {\n      console.log(`Image passed moderation: ${fileName}`);\n    }\n  } catch (error) {\n    console.error(`Error scanning image: ${fileName}`, error);\n  }\n};\n\n","js",[102,348,349,355,360,365,370,401,420,438,457,463,500,539,551,563,568,576,582,627,640,645,654,672,685,697,709,721,731,737,743,769,774,780,786,792,798,804,824,835,853,858,870,891,896],{"__ignoreMap":104},[146,350,351],{"class":148,"line":149},[146,352,354],{"class":353},"sJ8bj","\u002F\u002F This file is used for setting up a cloud function via the gcloud CLI.\n",[146,356,357],{"class":148,"line":156},[146,358,359],{"class":353},"\u002F\u002F I used it to set up the function to be triggered by a Pub\u002FSub message,\n",[146,361,362],{"class":148,"line":166},[146,363,364],{"class":353},"\u002F\u002F specifically the message that is triggered by a new file being uploaded to\n",[146,366,367],{"class":148,"line":172},[146,368,369],{"class":353},"\u002F\u002F a GCS bucket.\n",[146,371,372,376,379,382,385,388,392,395,398],{"class":148,"line":181},[146,373,375],{"class":374},"szBVR","const",[146,377,378],{"class":152}," { ",[146,380,381],{"class":159},"Storage",[146,383,384],{"class":152}," } ",[146,386,387],{"class":374},"=",[146,389,391],{"class":390},"sScJk"," require",[146,393,394],{"class":152},"(",[146,396,397],{"class":190},"'@google-cloud\u002Fstorage'",[146,399,400],{"class":152},");\n",[146,402,403,405,408,411,413,415,418],{"class":148,"line":197},[146,404,375],{"class":374},[146,406,407],{"class":159}," vision",[146,409,410],{"class":374}," =",[146,412,391],{"class":390},[146,414,394],{"class":152},[146,416,417],{"class":190},"'@google-cloud\u002Fvision'",[146,419,400],{"class":152},[146,421,422,424,427,429,432,435],{"class":148,"line":210},[146,423,375],{"class":374},[146,425,426],{"class":159}," storage",[146,428,410],{"class":374},[146,430,431],{"class":374}," new",[146,433,434],{"class":390}," Storage",[146,436,437],{"class":152},"();\n",[146,439,440,442,445,447,449,452,455],{"class":148,"line":222},[146,441,375],{"class":374},[146,443,444],{"class":159}," client",[146,446,410],{"class":374},[146,448,431],{"class":374},[146,450,451],{"class":152}," vision.",[146,453,454],{"class":390},"ImageAnnotatorClient",[146,456,437],{"class":152},[146,458,459],{"class":148,"line":235},[146,460,462],{"emptyLinePlaceholder":461},true,"\n",[146,464,465,468,471,473,475,478,481,485,488,491,494,497],{"class":148,"line":246},[146,466,467],{"class":159},"exports",[146,469,470],{"class":152},".",[146,472,119],{"class":390},[146,474,410],{"class":374},[146,476,477],{"class":374}," async",[146,479,480],{"class":152}," (",[146,482,484],{"class":483},"s4XuR","event",[146,486,487],{"class":152},", ",[146,489,490],{"class":483},"context",[146,492,493],{"class":152},") ",[146,495,496],{"class":374},"=>",[146,498,499],{"class":152}," {\n",[146,501,502,505,508,510,513,515,518,521,524,527,530,533,536],{"class":148,"line":252},[146,503,504],{"class":374},"  const",[146,506,507],{"class":159}," data",[146,509,410],{"class":374},[146,511,512],{"class":159}," JSON",[146,514,470],{"class":152},[146,516,517],{"class":390},"parse",[146,519,520],{"class":152},"(Buffer.",[146,522,523],{"class":390},"from",[146,525,526],{"class":152},"(event.data, ",[146,528,529],{"class":190},"'base64'",[146,531,532],{"class":152},").",[146,534,535],{"class":390},"toString",[146,537,538],{"class":152},"());\n",[146,540,541,543,546,548],{"class":148,"line":258},[146,542,504],{"class":374},[146,544,545],{"class":159}," bucketName",[146,547,410],{"class":374},[146,549,550],{"class":152}," data.bucket;\n",[146,552,553,555,558,560],{"class":148,"line":264},[146,554,504],{"class":374},[146,556,557],{"class":159}," fileName",[146,559,410],{"class":374},[146,561,562],{"class":152}," data.name;\n",[146,564,566],{"class":148,"line":565},14,[146,567,462],{"emptyLinePlaceholder":461},[146,569,571,574],{"class":148,"line":570},15,[146,572,573],{"class":374},"  try",[146,575,499],{"class":152},[146,577,579],{"class":148,"line":578},16,[146,580,581],{"class":353},"    \u002F\u002F This is where we have the newly uploaded image scanned by Google SafeSearch.\n",[146,583,585,588,591,594,597,599,602,605,608,610,613,616,619,622,625],{"class":148,"line":584},17,[146,586,587],{"class":374},"    const",[146,589,590],{"class":152}," [",[146,592,593],{"class":159},"result",[146,595,596],{"class":152},"] ",[146,598,387],{"class":374},[146,600,601],{"class":374}," await",[146,603,604],{"class":152}," client.",[146,606,607],{"class":390},"safeSearchDetection",[146,609,394],{"class":152},[146,611,612],{"class":190},"`gs:\u002F\u002F${",[146,614,615],{"class":152},"bucketName",[146,617,618],{"class":190},"}\u002F${",[146,620,621],{"class":152},"fileName",[146,623,624],{"class":190},"}`",[146,626,400],{"class":152},[146,628,630,632,635,637],{"class":148,"line":629},18,[146,631,587],{"class":374},[146,633,634],{"class":159}," safeSearch",[146,636,410],{"class":374},[146,638,639],{"class":152}," result.safeSearchAnnotation;\n",[146,641,643],{"class":148,"line":642},19,[146,644,462],{"emptyLinePlaceholder":461},[146,646,648,651],{"class":148,"line":647},20,[146,649,650],{"class":374},"    if",[146,652,653],{"class":152}," (\n",[146,655,657,660,663,666,669],{"class":148,"line":656},21,[146,658,659],{"class":152},"        safeSearch.adult ",[146,661,662],{"class":374},"===",[146,664,665],{"class":190}," 'VERY_LIKELY'",[146,667,668],{"class":374}," ||",[146,670,671],{"class":152}," \n",[146,673,675,677,679,682],{"class":148,"line":674},22,[146,676,659],{"class":152},[146,678,662],{"class":374},[146,680,681],{"class":190}," 'LIKELY'",[146,683,684],{"class":374}," ||\n",[146,686,688,691,693,695],{"class":148,"line":687},23,[146,689,690],{"class":152},"        safeSearch.racy ",[146,692,662],{"class":374},[146,694,665],{"class":190},[146,696,684],{"class":374},[146,698,700,703,705,707],{"class":148,"line":699},24,[146,701,702],{"class":152},"        safeSearch.medical ",[146,704,662],{"class":374},[146,706,665],{"class":190},[146,708,684],{"class":374},[146,710,712,715,717,719],{"class":148,"line":711},25,[146,713,714],{"class":152},"        safeSearch.violence ",[146,716,662],{"class":374},[146,718,665],{"class":190},[146,720,684],{"class":374},[146,722,724,726,728],{"class":148,"line":723},26,[146,725,714],{"class":152},[146,727,662],{"class":374},[146,729,730],{"class":190}," 'LIKELY'\n",[146,732,734],{"class":148,"line":733},27,[146,735,736],{"class":152},"    ) {\n",[146,738,740],{"class":148,"line":739},28,[146,741,742],{"class":353},"      \u002F\u002F And this is where we delete an image that doesn't pass the check.\n",[146,744,746,749,752,755,758,761,764,767],{"class":148,"line":745},29,[146,747,748],{"class":374},"      await",[146,750,751],{"class":152}," storage.",[146,753,754],{"class":390},"bucket",[146,756,757],{"class":152},"(bucketName).",[146,759,760],{"class":390},"file",[146,762,763],{"class":152},"(fileName).",[146,765,766],{"class":390},"delete",[146,768,437],{"class":152},[146,770,772],{"class":148,"line":771},30,[146,773,462],{"emptyLinePlaceholder":461},[146,775,777],{"class":148,"line":776},31,[146,778,779],{"class":353},"      \u002F\u002F These logs show in the Google cloud console whenever the cloud function\n",[146,781,783],{"class":148,"line":782},32,[146,784,785],{"class":353},"      \u002F\u002F is triggered.\n",[146,787,789],{"class":148,"line":788},33,[146,790,791],{"class":353},"      \u002F\u002F In the future I will also update this area of the code so that it triggers a \n",[146,793,795],{"class":148,"line":794},34,[146,796,797],{"class":353},"      \u002F\u002F notification, which will tell the user that their image was deleted because it didn't \n",[146,799,801],{"class":148,"line":800},35,[146,802,803],{"class":353},"      \u002F\u002F pass the content filter.\n",[146,805,807,810,813,815,818,820,822],{"class":148,"line":806},36,[146,808,809],{"class":152},"      console.",[146,811,812],{"class":390},"log",[146,814,394],{"class":152},[146,816,817],{"class":190},"`Deleted unsafe image: ${",[146,819,621],{"class":152},[146,821,624],{"class":190},[146,823,400],{"class":152},[146,825,827,830,833],{"class":148,"line":826},37,[146,828,829],{"class":152},"    } ",[146,831,832],{"class":374},"else",[146,834,499],{"class":152},[146,836,838,840,842,844,847,849,851],{"class":148,"line":837},38,[146,839,809],{"class":152},[146,841,812],{"class":390},[146,843,394],{"class":152},[146,845,846],{"class":190},"`Image passed moderation: ${",[146,848,621],{"class":152},[146,850,624],{"class":190},[146,852,400],{"class":152},[146,854,856],{"class":148,"line":855},39,[146,857,255],{"class":152},[146,859,861,864,867],{"class":148,"line":860},40,[146,862,863],{"class":152},"  } ",[146,865,866],{"class":374},"catch",[146,868,869],{"class":152}," (error) {\n",[146,871,873,876,879,881,884,886,888],{"class":148,"line":872},41,[146,874,875],{"class":152},"    console.",[146,877,878],{"class":390},"error",[146,880,394],{"class":152},[146,882,883],{"class":190},"`Error scanning image: ${",[146,885,621],{"class":152},[146,887,624],{"class":190},[146,889,890],{"class":152},", error);\n",[146,892,894],{"class":148,"line":893},42,[146,895,334],{"class":152},[146,897,899],{"class":148,"line":898},43,[146,900,901],{"class":152},"};\n",[60,903,905],{"id":904},"deploying-the-function","Deploying the Function",[15,907,908],{},"Then, with my terminal in the same directory as my new cloud function code, I deployed the cloud function with this command:",[95,910,913],{"className":911,"code":912,"language":100},[98],"gcloud functions deploy scanImage \\\n  --runtime nodejs20 \\\n  --trigger-topic=image-uploads-topic \\\n  --entry-point=scanImage \\\n  --allow-unauthenticated\n",[102,914,912],{"__ignoreMap":104},[10,916,918],{"id":917},"testing-the-content-filter","Testing the Content Filter",[15,920,921],{},"I was glad to learn that this approach was possible, because from a technical or architectural perspective, it's nifty. With only four commands and a few lines of code, my app already has a basic level of protection from vandalism. The best thing about it is that I never had to change a single line of code in my app, because the content filter is handled entirely on the GCP side.",[15,923,924],{},"Then there was only one thing left to do: test it to make sure that it works.",[60,926,928],{"id":927},"understanding-safesearch-labels","Understanding SafeSearch Labels",[15,930,931],{},"Now, if you're like me, when you see that SafeSearch returns a response like this, you have questions:",[95,933,936],{"className":934,"code":141,"language":935,"meta":104,"style":104},"language-JSON shiki shiki-themes github-light github-dark","JSON",[102,937,938,942,948,952,958,968,978,988,998,1006,1010,1014,1018],{"__ignoreMap":104},[146,939,940],{"class":148,"line":149},[146,941,153],{"class":152},[146,943,944,946],{"class":148,"line":156},[146,945,160],{"class":159},[146,947,163],{"class":152},[146,949,950],{"class":148,"line":166},[146,951,169],{"class":152},[146,953,954,956],{"class":148,"line":172},[146,955,175],{"class":159},[146,957,178],{"class":152},[146,959,960,962,964,966],{"class":148,"line":181},[146,961,184],{"class":159},[146,963,187],{"class":152},[146,965,191],{"class":190},[146,967,194],{"class":152},[146,969,970,972,974,976],{"class":148,"line":197},[146,971,200],{"class":159},[146,973,187],{"class":152},[146,975,205],{"class":190},[146,977,194],{"class":152},[146,979,980,982,984,986],{"class":148,"line":210},[146,981,213],{"class":159},[146,983,187],{"class":152},[146,985,205],{"class":190},[146,987,194],{"class":152},[146,989,990,992,994,996],{"class":148,"line":222},[146,991,225],{"class":159},[146,993,187],{"class":152},[146,995,230],{"class":190},[146,997,194],{"class":152},[146,999,1000,1002,1004],{"class":148,"line":235},[146,1001,238],{"class":159},[146,1003,187],{"class":152},[146,1005,243],{"class":190},[146,1007,1008],{"class":148,"line":246},[146,1009,249],{"class":152},[146,1011,1012],{"class":148,"line":252},[146,1013,255],{"class":152},[146,1015,1016],{"class":148,"line":258},[146,1017,261],{"class":152},[146,1019,1020],{"class":148,"line":264},[146,1021,267],{"class":152},[15,1023,1024],{},"The top question on my mind was, \"Where does it draw the line? What's the difference between 'LIKELY' and 'VERY_LIKELY' racy?\"",[15,1026,1027,1028,1033],{},"I checked the reference documentation for SafeSearchAnnotation, which is located ",[52,1029,1032],{"href":1030,"rel":1031},"https:\u002F\u002Fcloud.google.com\u002Fvision\u002Fdocs\u002Freference\u002Frest\u002Fv1\u002FAnnotateImageResponse#safesearchannotation",[56],"here."," As you can see, the reference material is fairly sparse. It defines \"racy\" as the following:",[1035,1036,1037],"blockquote",{},[15,1038,1039],{},"Likelihood that the request image contains racy content. Racy content may include (but is not limited to) skimpy or sheer clothing, strategically covered nudity, lewd or provocative poses, or close-ups of sensitive body areas.",[60,1041,1043],{"id":1042},"almost-real-world-testing","(Almost) Real-World Testing",[15,1045,1046],{},"These reference materials did not contain what I was really looking for, which was a helpful chart with three columns - POSSIBLE, LIKELY and VERY_LIKELY - with rows for adult, spoof, medical, violence and racy content, with an example picture in each table cell.",[15,1048,1049],{},"I wonder why they did not include that???",[15,1051,1052],{},"Jokes aside, it became clear that the only way to really understand those labels is to test it with our own images. So in the following test, I wanted to both a) check if the new filter could successfully remove images and b) get an idea of where the dividing line is between different levels of raciness.",[15,1054,1055,1056,1061],{},"So I decided to test the filter with racy content. I felt uncomfortable about using pictures of real people for the purpose of testing my content filter, so I decided this would be a good time to use ",[52,1057,1060],{"href":1058,"rel":1059},"https:\u002F\u002Fx.com\u002Fi\u002Fgrok?focus=1",[56],"Grok"," to generate racy images of fake people as test data.",[15,1063,1064],{},"After requesting these AI images from Grok, I had two pictures in my test data. Test picture 1 is somewhat of a milder level of raciness, while test picture 2 is more explicit.",[15,1066,1067],{},"Test picture 1:",[15,1069,1070],{},[1071,1072],"img",{"alt":1073,"src":1074},"Woman posing while wearing lingerie","\u002Fposts\u002Fbasic-content-filter\u002Ftest-picture-1.jpg",[15,1076,1077],{},"Test picture 2:",[15,1079,1080],{},[146,1081,1082],{},"The second test picture was of a similar photo, in which a woman is also wearing lingerie, while her hand was implied to be doing a sexual act outside of the frame.",[15,1084,1085,1086,1089,1090,1093],{},"To test it, I ran my database locally, created a forum ",[102,1087,1088],{},"image_test"," and a test post called ",[102,1091,1092],{},"Image test."," In Multiforum, the text editor is similar to the text editor in GitHub, which lets you drag and drop images into the text editor to upload them, so that you can easily interleave images in text. So to do this, I dragged the test pictures 1 and 2 into the text editor, and you can see here that they were uploaded because the new image URLs are included there in the markdown:",[15,1095,1096],{},[1071,1097],{"alt":1098,"src":1099},"Text editor with URLs in markdown, indicating successful upload","\u002Fposts\u002Fbasic-content-filter\u002Fdoing-the-image-test.png",[10,1101,1103],{"id":1102},"results-and-conclusion","Results and Conclusion",[15,1105,1106],{},"Then I saved the post, and voilà! Test picture 1 passed the filter, while test picture 2 failed the content check, and it got automatically deleted even before I clicked the save button:",[15,1108,1109],{},[1071,1110],{"alt":1111,"src":1112},"Post with one image showing successfully and one filename of an image that was deleted","\u002Fposts\u002Fbasic-content-filter\u002Fcontent-filter-success.png",[15,1114,1115,1116,1118,1119,470],{},"On top of that, I can confirm that it worked by checking the logs in the Google Cloud Platform console, in which I can see that the ",[102,1117,119],{}," cloud function logged ",[102,1120,1121],{},"Deleted unsafe image: 1736991660484-cluse-explicit-test.jpg",[15,1123,1124],{},"And that's it. Huge success. In the future, when an image is deleted because of this automated check, I'll send a notification to the user who uploaded it, saying \"This image was automatically removed by an AI content filter. AI makes mistakes, so if you think this decision was made in error, please open a support ticket\" or something along those lines.",[15,1126,1127],{},"For now, I'm okay with the line being drawn somewhere in between those two test pictures. At least now the line is being drawn somewhere.",[1129,1130,1131],"style",{},"html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}",{"title":104,"searchDepth":156,"depth":156,"links":1133},[1134,1135,1136,1140,1147,1151],{"id":12,"depth":156,"text":13},{"id":26,"depth":156,"text":27},{"id":46,"depth":156,"text":47,"children":1137},[1138,1139],{"id":62,"depth":166,"text":63},{"id":72,"depth":166,"text":73},{"id":82,"depth":156,"text":83,"children":1141},[1142,1143,1144,1145,1146],{"id":89,"depth":166,"text":90},{"id":133,"depth":166,"text":134},{"id":283,"depth":166,"text":284},{"id":337,"depth":166,"text":338},{"id":904,"depth":166,"text":905},{"id":917,"depth":156,"text":918,"children":1148},[1149,1150],{"id":927,"depth":166,"text":928},{"id":1042,"depth":166,"text":1043},{"id":1102,"depth":156,"text":1103},"2025-01-16T21:25:10.072Z","How I added image moderation to Multiforum using Google Cloud storage events, Cloud Run, and the Vision API.","md",{},"\u002Fpost\u002Fbasic-content-filter",{"title":5,"description":1153},"post\u002Fbasic-content-filter","7Ep9ygrORONLIFdNTpWEWjvC0QbOeK3OB0kKE17JzIc",1785377801502]