Kanye worried they're going to tale his family away from him