Erratic issue: Variables sometimes not properly passed into templates

[stucka]

Sometimes I can generate a project and it'll work just fine. Minutes later, part of it fails, unpredictably, at different points in the process. The variables don't populate into the template.

I don't know why I'm getting these failures, but a workaround is to insert with a filter like
{{ variablename | dieifblank }}. They seem to fail as type None.

The working code:

def filter_dieifblank(s):
    from sys import exit
    if not s:
        print("!!!!!!!!!!\r\n!!!!!!!!!!!\r\n!!!!!!!!!!!!\r\n!!!!!!!!!!!!!\r\r!!!!!!!!!!!!!!!!!!")
        print("We got a None value where you wanted substance.")
        exit()
    elif len(s) == 0:
        print("!!!!!!!!!!\r\n!!!!!!!!!!!\r\n!!!!!!!!!!!!\r\n!!!!!!!!!!!!!\r\r!!!!!!!!!!!!!!!!!!")
        print("We got an empty but defined string")
        exit()
    else:    
        return s```

[eyeseast]

You might need to show more of your code so I can reproduce this. Never had anything like that happen before.

Are the variables in question being populated from a spreadsheet, or from somewhere in tarbell_config.py?

[stucka]

Generated in tarbell_config.py. Ryan Marx seemed to think he'd had a similar problem:
https://newsnerdery.slack.com/archives/C08B2APFX/p1518636225000100

The code is ... uh ... kinda copypasta'd after several iterations and spinoffs from other projects, but it's staying the same. Here's the whole section:

@blueprint.before_request
def find_photos():
    import os
    site = g.current_site
    context = site.get_context()
#    if not "gotphotos" in context:    # Let's run this just once.
# And this dies on live Tarbell, but not production. So ...
    if 1 != 2:
        context["gotphotos"] = "Got photos!"
        imgpath = "img/fullres/"
        targetpath = "img/candidates/"
        defaultimg = "img/extras/head-outline.png"
        desiredsizes = [201]        # Just one size of thumbnail. Not worth complexity otherwise.
        photolist = os.listdir(imgpath)
        for i, photo in enumerate(photolist):
            photolist[i] = str(photo)[:-4]   # knock off file extension
        namekeylist = []
        # pp = pprint.PrettyPrinter(indent=4)
        # pp.pprint(context)
        for i, candidate in enumerate(context['candidates']):
            namekeylist.append(candidate['namekey'])        # generate a list of namekeys
            targetfilename = imgpath + candidate['namekey'] + ".jpg"
            if os.path.exists(targetfilename) and candidate['status'] == "received":    # If we have photo but no survey, we don't use photo
                context['candidates'][i]["photofilename"] = targetpath + candidate['namekey']
                photolist.remove(candidate['namekey'])
                for targetwidth in desiredsizes:
                    thumbnailname = targetpath + candidate['namekey'] + str(targetwidth) + ".jpg"
                    if not os.path.exists(thumbnailname):
                        im = Image.open(targetfilename)
                        width, height = im.size
                        ratio = float(float(targetwidth)/float(width))
                        newwidth = int(round(width*ratio))
                        newheight = int(round(height*ratio))
                        im.resize((newwidth, newheight), Image.LANCZOS).save(thumbnailname, quality=70, optimize=True)
                    context['candidates'][i]["photofilename"] = thumbnailname
            else:
                context['candidates'][i]["photofilename"] = defaultimg
        if len(photolist) > 0:  # We have orphaned photos.
            print("Orphaned photos found: " + ", ".join(photolist))
            for orphan in photolist:
                best_name, best_score = list_match(orphan, namekeylist)
                print("\t" + orphan + "\t" + "Did you mean this?: " + best_name)
        else:
            print("No orphaned photos found")```

[eyeseast]

A couple things I can think of:

If it works when you build, but not when running the dev server, it's probably a state problem. Something is getting modified between requests, and that's breaking or unsetting your variables. You really want every request to start with a fresh context.

Case in point: site.get_context() is a problem child. It sets site.data which can then be modified. Setting anything on it in before_request (which should really be before_app_request) can cause stuff to leak between requests.

The other thing to look for is generators that will get used up in one request and not be available on the next. Make sure if you're running anything that calls yield it gets created fresh for each request.

[stucka]

I'll give this a look -- thank you.

I think I'd had the bug running the dev server, but refreshing fixed it, so I moved on. I hadn't realized the extent of this until recently.

[stucka]

I was still experiencing this, and it intensified on a recent project.

Tried troubleshooting a bunch of different ways, but @eyeseast 's comments about the context got me oriented. I think it's a problem when the Google sheet gets hit too hard.

Setting the cache to outlast the baketime of my project seems to fix it. Before I was getting three or four errors per run, and now have two runs complete with zero errors:
SPREADSHEET_CACHE_TTL = 150

My method of testing for a failed variable was to look for blank entries and error out. The template:
<img class="candidatephoto" src="{{homepage}}/{{selectedstatus['photofilename'] | dieifblank }}">

@blueprint.app_template_filter('dieifblank')
def filter_dieifblank(s):
    from sys import exit
    if not s:
        print("!!!!!!!!!!\r\n!!!!!!!!!!!\r\n!!!!!!!!!!!!\r\n!!!!!!!!!!!!!\r\r!!!!!!!!!!!!!!!!!!")
        print("We got a None value where you wanted substance.")
        exit() ### HEYYYYYYYYYYYYYYYYYYYYYYYYYY! HEY!
    elif len(s) == 0:
        print("!!!!!!!!!!\r\n!!!!!!!!!!!\r\n!!!!!!!!!!!!\r\n!!!!!!!!!!!!!\r\r!!!!!!!!!!!!!!!!!!")
        print("We got an empty but defined string")
        exit() ### HEYYYYYYYYYYYYYYYYYYYYYYYYYY! HEY!
    else:    
        return s

I think it was always showing the None value.