Is this even possible in AppGameKit? Maybe with a shader?
Yes. In fact you can do anything a professional editing software can do with shaders.
this is a simple agk greenscreen pixelshader
you need to render the video to an image as an input and apply the pixelshader and image to a sprite
+ Code Snippet#define threshold 0.55
#define padding 0.05
uniform sampler2D texture0;
varying mediump vec2 uvVarying;
varying mediump vec4 colorVarying;
void main()
{
vec4 greenScreen = vec4(0.,1.,0.,1.);
vec4 color = texture2D(texture0, uvVarying);
vec3 diff = color.xyz - greenScreen.xyz;
float fac = smoothstep(threshold-padding,threshold+padding, dot(diff,diff));
color = mix(color,texture2D(texture0, uvVarying) * colorVarying, 1.-fac);
gl_FragColor = color;
}